Data storage integrity validation

ABSTRACT

Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application incorporates by reference for all purposes the fulldisclosure of co-pending U.S. patent application Ser. No. 13/569,984,filed concurrently herewith, entitled “LOG-BASED DATA STORAGE ONSEQUENTIALLY WRITTEN MEDIA” (Attorney Docket No. 90204-841804(054800US)), co-pending U.S. patent application Ser. No. 13/570,057,filed concurrently herewith, entitled “DATA STORAGE MANAGEMENT FORSEQUENTIALLY WRITTEN MEDIA” (Attorney Docket No. 90204-841817(055300US)), co-pending U.S. patent application Ser. No. 13/570,005,filed concurrently herewith, entitled “DATA WRITE CACHING FORSEQUENTIALLY WRITTEN MEDIA” (Attorney Docket No. 90204-841812(055000US)), co-pending U.S. patent application Ser. No. 13/570,030,filed concurrently herewith, entitled “PROGRAMMABLE CHECKSUMCALCULATIONS ON DATA STORAGE DEVICES” (Attorney Docket No. 90204-841813(055200US)), co-pending U.S. patent application Ser. No. 13/569,994,filed concurrently herewith, entitled “ARCHIVAL DATA IDENTIFICATION”(Attorney Docket No. 90204-841807 (054300US)), co-pending U.S. patentapplication Ser. No. 13/570,029, filed concurrently herewith, entitled“ARCHIVAL DATA ORGANIZATION AND MANAGEMENT” (Attorney Docket No.90204-841808 (054400US)), co-pending U.S. patent application Ser. No.13/570,092, filed concurrently herewith, entitled “ARCHIVAL DATA FLOWMANAGEMENT” (Attorney Docket No. 90204-841809 (054500US)), co-pendingU.S. patent application Ser. No. 13/570,088, filed concurrentlyherewith, entitled “ARCHIVAL DATA STORAGE SYSTEM” (Attorney Docket No.90204-841806 (054000US)), co-pending U.S. patent application Ser. No.13/569,591, filed concurrently herewith, entitled “DATA STORAGE POWERMANAGEMENT” (Attorney Docket No. 90204-841816 (054900US)), co-pendingU.S. patent application Ser. No. 13/569,714, filed concurrentlyherewith, entitled “DATA STORAGE SPACE MANAGEMENT” (Attorney Docket No.90204-846202 (056100US)), co-pending U.S. patent application Ser. No.13/570,074, filed concurrently herewith, entitled “DATA STORAGEAPPLICATION PROGRAMMING INTERFACE” (Attorney Docket No. 90204-846378(056200US)), and co-pending U.S. patent application Ser. No. 13/569,665,filed concurrently herewith, entitled “DATA STORAGE INVENTORY INDEXING”(Attorney Docket No. 90204-841811 (054700US)).

BACKGROUND

As more and more information is converted to digital form, the demandfor durable and reliable data storage services is ever increasing. Inparticular, archive records, backup files, media files and the like maybe maintained or otherwise managed by government entities, businesses,libraries, individuals, etc. However, the storage of digitalinformation, especially for long periods of time, has presented somechallenges. In some cases, the cost of long-term data storage may beprohibitive to many because of the potentially massive amounts of datato be stored, particularly when considering archival or backup data.Additionally, durability and reliability issues may be difficult tosolve for such large amounts of data and/or for data that is expected tobe stored for relatively long periods of time. Magnetic tapes havetraditionally been used in data backup systems because of the low cost.However, tape-based storage systems have been unable to fully exploitstorage technology advances. Additionally, drive-based storage systemsmay have difficulty in validating data integrity without preservingartifacts of information about how the data was broken up during upload.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an example flow for describing an implementation ofthe data integrity validation described herein, according to at leastone example.

FIG. 2 illustrates an example environment in which archival data storageservices may be implemented, in accordance with at least one embodiment.

FIG. 3 illustrates an interconnection network in which components of anarchival data storage system may be connected, in accordance with atleast one embodiment.

FIG. 4 illustrates an interconnection network in which components of anarchival data storage system may be connected, in accordance with atleast one embodiment.

FIG. 5 illustrates an example process for storing data, in accordancewith at least one embodiment.

FIG. 6 illustrates an example process for retrieving data, in accordancewith at least one embodiment.

FIG. 7 illustrates an example process for deleting data, in accordancewith at least one embodiment.

FIGS. 8-9 illustrate block diagrams for describing at least somefeatures of the data integrity validation described here, according toat least some examples.

FIGS. 10-12 illustrate example flow diagrams of one or more processesfor implementing at least some features of the data integrity validationdescribed herein, according to at least some examples.

FIG. 13 illustrates an environment in which various embodiments can beimplemented.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

Embodiments of the present disclosure are directed to, among otherthings, validating or otherwise verifying the integrity of data payloadsor portions of data payloads intended for storage. In some examples, anarchival data storage service may be configured to receive requests tostore data, in some cases large amounts of data, in logical datacontainers or other archival storage devices for varying, and oftenrelatively long, periods of time. In some examples, the archival datastorage service may operate or otherwise utilize many different storagedevices. For example, the archival storage service may include one ormore disk drives that, when operating, utilize spinning magnetic media.Additionally, the archival data storage service may include one or moreracks located in one or more geographic locations (e.g., potentiallyassociated to different postal codes). Each rack may further include oneor more, or even hundreds of, hard drives such as, but not limited to,disk drives or the like.

In some aspects, the archival data storage service may provide storage,access and/or placement of one or more computing resources through aservice such as, but not limited to, a web service, a remote programexecution service or other network-based data management service. Forexample, a user, client entity, computing resource or other computingdevice may access, via the archival data storage service, data storageand/or management such that access mechanisms may be implemented and/orprovided to the user or computing device. In some examples, computingresource services, such as those provided by the archival data storageservice, may include one or more computing resources accessible acrossone or more networks through user interfaces (UIs), applicationprogramming interfaces (APIs) and/or other interfaces where the one ormore computing resources may be scalable and/or expandable as desired.

In some examples, the archival data storage service may enable users orclient entities such as, but not limited to, third-party services thatutilize the archival data storage service or other web servicesassociated with the archival data storage service to upload data forpotentially long and persistent storage. Unless otherwise contradictedexplicitly or clearly by context, the term “user” is used herein todescribe any entity utilizing the archival data storage service. Thearchival data storage service may also wish to ensure the integrity ofthe archived data and/or guarantee the integrity of the archived data.In order to accomplish these goals, in some examples, the archival datastorage service may provide a data object identifier for identifying thedata once uploaded. In some cases, the data object identifier may alsoinclude a top-level tree digest for validating the archived data evenafter some extended period of time. Additionally, the data objectidentifier may also be configured in such a way that its integrity maybe validated as well.

The archival data storage service may be configured to performoperations on data to be stored, such that the data is broken intoparts, portions or other demarcated groupings. Once separated intoparts, the archival data storage service may perform one or moreencryption functions and/or algorithms on the parts including, but notlimited to, a hash function or other cryptographic or other method toproduce a digest (referred to also as a hash value, a hash code, achecksum, etc.). In some examples, the data may be validated such thatthe archival storage service can ensure that the data being storedmatches the data received. In this way, data integrity may be validated.Additionally, the sender of the data may be able to independentlydetermine a data chunk size (i.e., the size of the parts of the payload)and not be requested to maintain or persist this information. Forexample, a user or client entity may request to upload a 1 gigabyte (GB)file in 2 megabyte (MB) portions (chunks). And without saving orotherwise persisting the fact that 2 MB chunks were used, the datapayload may still be validated. That is, based at least in part ongenerating one or more digests, checksums, hash codes, etc., for thechunks and at least a digest for the payload (in some examples based atleast in part on combinations of the digests), the data may later bepartitioned into different sized chunks without losing the ability to bevalidated.

In some aspects, this may be accomplished by providing instructions oran algorithm to a user or client entity that indicates the way in whichthe data should be partitioned and/or hashed. Additionally, in somecases, the archival storage service may expose or otherwise provide oneor more API method calls and/or a software development kit (SDK) toenable users to appropriately upload the data in chunks and tofacilitate the requested order of operations and/or inclusion ofappropriate checksum information. For example, the users may berequested to select a chunk size of at least 1 MB, or some otherpredefined size, for uploading the data. As used herein, the data beinguploaded by a user or client entity and/or stored by the archival datastorage service may be referred to as a payload. Additionally, the usermay select or otherwise instruct the archival data storage service thatthe payload is to be partitioned into sizes including powers of two of 1MB (e.g., 1 MB, 2 MB, 4 MB, 8 MB, 16 MB, etc.). In other examples, thepayload may be partitioned into sizes including other multiples of apredefined size (e.g., 1 MB). The other multiples may be based at leastin part on the degree of children of a tree used to represent the data.For example, if a binary tree is used, the integer multiple may includeinteger powers of two, if a trinary tree is used, the integer multiplemay be integer powers of three, if a tree of degree four (i.e., eachnode may have four children) is used, the integer multiple may includeinteger powers of four, and so on. The user may then follow an algorithmfor generating one or more hash trees (e.g., one hash tree per part whenthe payload is partitioned) and/or one or more hash values for eachpartition. Additionally, the user may generate a hash value (or digest)for each 1 MB chunk independent of the selected partition size. In someexamples, these digests corresponding to the 1 MB chunks of a partitionmay be included in a hash tree for the partition. Further, in someexamples, a root node of each partition's hash tree may be provided tothe archival data storage service along with the digests for each 1 MBsub-part of the partition. In this way, each sub-part may be validatedby the archival data storage service, based at least in part on the 1 MBdigests, and each partition may be validated by the archival datastorage service, based at least in part on the root digest for eachpart. Additionally, a final root hash associated with a hash of eachroot hash for the parts may be used by the archival data storageservice, based at least in part on comparing the received final roothash and a top-level hash value determined by the archival data storageservice.

In at least one example, generating a hash tree or other hierarchicaldata structure (e.g., other types of binary trees such as, but notlimited to, B-trees and/or other types of data structures such as, butnot limited to, arrays, records, etc.) may include concatenating digestsand running a hash function on the concatenated digest. For example, ina binary hash tree, a root node may have two children represented by onehash value each. In some cases, generating the root hash value may bebased at least in part on concatenating the two children hash values toform a new piece of data and further running the hash function on thenew piece of data. The resulting hash value may represent the root hash.As such, each partition of a payload may have its own root hash,although its root hash may be used in calculating the top-level hash forthe payload. In some examples, it may also be possible to validate thepayload and/or portions of the payload without recalling or otherwisepersisting the partition size chosen by the user.

FIG. 1 depicts an illustrative flow 100 in which techniques for thevalidation of data integrity may be implemented. These techniques aredescribed in more detail below in connection with FIGS. 8-12. Returningto FIG. 1, in illustrative flow 100, operations may be performed by oneor more processors of an archival data storage service and/orinstructions that for performing the operations may be stored in one ormore memories of the archival data storage service. As desired, the flow100 may begin at 102, where the archival data storage service mayreceive one or more parts of a data payload 104. In some examples, thedata payload 104 may include any number of parts; however, in thisexample two parts are shown, Part 1 and Part 2. Each of Part 1 and Part2 may include data 106 and 108, respectively. In some cases, the size ofPart 1 and Part 2 may be selected by the uploader and/or may be thesame. However, in some examples, the last part of a data payload 104 maybe a different size from all the other consistently sized parts (e.g.,as shown here in FIG. 1, wherein Part 1 is bigger than Part 2). At 110,the flow 100 may generate sub-parts of the parts of the payload 104. Insome examples, the size of the sub-parts may be predefined by thearchival data storage service (e.g., 1 MB).

In some examples, the flow 100 may calculate a digest for each sub-partat 112. The respective digests may be stored as nodes 114 of a datastructure such as, but not limited to, the data structure 116 generatedat 118. By way of example only, the data structure 116 may include oneor more sub-part digests (e.g., at nodes 114) and/or one or more partdigests (e.g., Part 1 digest 120 and Part 2 digest 122). Additionally,at 124, the flow 100 may determine a root digest 126 for the root of thedata structure 116. In some examples, the root digest 126 may bedetermined or generated based at least in part on concatenating partdigests and calculating a digest for the concatenated digests. The flow100 may end at 118, where the archival data storage service may verifythat the received payload 104 matches a stored payload 130. The storedpayload may, in some examples, contain data 132 determined based atleast in part on combining each of the parts 106, 108 and/or sub-parts,when received. In some examples, verifying the data payload may be basedat least in part on comparing the root digest 126 against a second rootdigest received from the uploader.

FIG. 2 illustrates an example environment 200 in which an archival datastorage system may be implemented, in accordance with at least oneembodiment. One or more customers 202 connect, via a network 204, to anarchival data storage system 206. As implied above, unless otherwiseclear from context, the term “customer” refers to the system(s) of acustomer entity (such as an individual, company or other organization)that utilizes data storage services described herein. Such systems mayinclude datacenters, mainframes, individual computing devices,distributed computing environments and customer-accessible instancesthereof or any other system capable of communicating with the archivaldata storage system. In some embodiments, a customer may refer to amachine instance (e.g., with direct hardware access) or virtual instanceof a distributed computing system provided by a computing resourceprovider that also provides the archival data storage system. In someembodiments, the archival data storage system is integral to thedistributed computing system and may include or be implemented by aninstance, virtual or machine, of the distributed computing system. Invarious embodiments, network 204 may include the Internet, a local areanetwork (“LAN”), a wide area network (“WAN”), a cellular data networkand/or other data network.

In an embodiment, archival data storage system 206 provides amulti-tenant or multi-customer environment where each tenant or customermay store, retrieve, delete or otherwise manage data in a data storagespace allocated to the customer. In some embodiments, an archival datastorage system 206 comprises multiple subsystems or “planes” that eachprovides a particular set of services or functionalities. For example,as illustrated in FIG. 2, archival data storage system 206 includesfront end 208, control plane for direct I/O 210, common control plane212, data plane 214 and metadata plane 216. Each subsystem or plane maycomprise one or more components that collectively provide the particularset of functionalities. Each component may be implemented by one or morephysical and/or logical computing devices, such as computers, datastorage devices and the like. Components within each subsystem maycommunicate with components within the same subsystem, components inother subsystems or external entities such as customers. At least someof such interactions are indicated by arrows in FIG. 2. In particular,the main bulk data transfer paths in and out of archival data storagesystem 206 are denoted by bold arrows. It will be appreciated by thoseof ordinary skill in the art that various embodiments may have fewer ora greater number of systems, subsystems and/or subcomponents than areillustrated in FIG. 2. Thus, the depiction of environment 200 in FIG. 2should be taken as being illustrative in nature and not limiting to thescope of the disclosure.

In the illustrative embodiment, front end 208 implements a group ofservices that provides an interface between the archival data storagesystem 206 and external entities, such as one or more customers 202described herein. In various embodiments, front end 208 provides anapplication programming interface (“API”) to enable a user toprogrammatically interface with the various features, components andcapabilities of the archival data storage system. Such APIs may be partof a user interface that may include graphical user interfaces (GUIs),Web-based interfaces, programmatic interfaces such as applicationprogramming interfaces (APIs) and/or sets of remote procedure calls(RPCs) corresponding to interface elements, messaging interfaces inwhich the interface elements correspond to messages of a communicationprotocol, and/or suitable combinations thereof.

Capabilities provided by archival data storage system 206 may includedata storage, data retrieval, data deletion, metadata operations,configuration of various operational parameters and the like. Metadataoperations may include requests to retrieve catalogs of data stored fora particular customer, data recovery requests, job inquires and thelike. Configuration APIs may allow customers to configure accountinformation, audit logs, policies, notifications settings and the like.A customer may request the performance of any of the above operations bysending API requests to the archival data storage system. Similarly, thearchival data storage system may provide responses to customer requests.Such requests and responses may be submitted over any suitablecommunications protocol, such as Hypertext Transfer Protocol (“HTTP”),File Transfer Protocol (“FTP”) and the like, in any suitable format,such as REpresentational State Transfer (“REST”), Simple Object AccessProtocol (“SOAP”) and the like. The requests and responses may beencoded, for example, using Base64 encoding, encrypted with acryptographic key or the like.

In some embodiments, archival data storage system 206 allows customersto create one or more logical structures such as a logical datacontainers in which to store one or more archival data objects. As usedherein, data object is used broadly and does not necessarily imply anyparticular structure or relationship to other data. A data object maybe, for instance, simply a sequence of bits. Typically, such logicaldata structures may be created to meeting certain business requirementsof the customers and are independently of the physical organization ofdata stored in the archival data storage system. As used herein, theterm “logical data container” refers to a grouping of data objects. Forexample, data objects created for a specific purpose or during aspecific period of time may be stored in the same logical datacontainer. Each logical data container may include nested datacontainers or data objects and may be associated with a set of policiessuch as size limit of the container, maximum number of data objects thatmay be stored in the container, expiration date, access control list andthe like. In various embodiments, logical data containers may becreated, deleted or otherwise modified by customers via API requests, bya system administrator or by the data storage system, for example, basedon configurable information. For example, the following HTTP PUT requestmay be used, in an embodiment, to create a logical data container withname “logical-container-name” associated with a customer identified byan account identifier “accountId”.

PUT /{accountId}/logical-container-name HTTP/1.1

In an embodiment, archival data storage system 206 provides the APIs forcustomers to store data objects into logical data containers. Forexample, the following HTTP POST request may be used, in an illustrativeembodiment, to store a data object into a given logical container. In anembodiment, the request may specify the logical path of the storagelocation, data length, reference to the data payload, a digital digestof the data payload and other information. In one embodiment, the APIsmay allow a customer to upload multiple data objects to one or morelogical data containers in one request. In another embodiment where thedata object is large, the APIs may allow a customer to upload the dataobject in multiple parts, each with a portion of the data object.

POST /{accountId}/logical-container-name/data HTTP/1.1 Content-Length:1128192 x-ABC-data-description: ”annual-result-2012.xls”x-ABC-md5-tree-hash: 634d9a0688aff95c

In response to a data storage request, in an embodiment, archival datastorage system 206 provides a data object identifier if the data objectis stored successfully. Such data object identifier may be used toretrieve, delete or otherwise refer to the stored data object insubsequent requests. In some embodiments, such as data object identifiermay be “self-describing” in that it includes (for example, with orwithout encryption) storage location information that may be used by thearchival data storage system to locate the data object without the needfor an additional data structures such as a global namespace key map. Inaddition, in some embodiments, data object identifiers may also encodeother information such as payload digest, error-detection code, accesscontrol data and the other information that may be used to validatesubsequent requests and data integrity. In some embodiments, thearchival data storage system stores incoming data in a transient durabledata store before moving it archival data storage. Thus, althoughcustomers may perceive that data is persisted durably at the moment whenan upload request is completed, actual storage to a long-term persisteddata store may not commence until sometime later (e.g., 12 hours later).In some embodiments, the timing of the actual storage may depend on thesize of the data object, the system load during a diurnal cycle,configurable information such as a service-level agreement between acustomer and a storage service provider and other factors.

In some embodiments, archival data storage system 206 provides the APIsfor customers to retrieve data stored in the archival data storagesystem. In such embodiments, a customer may initiate a job to performthe data retrieval and may learn the completion of the job by anotification or by polling the system for the status of the job. As usedherein, a “job” refers to a data-related activity corresponding to acustomer request that may be performed temporally independently from thetime the request is received. For example, a job may include retrieving,storing and deleting data, retrieving metadata and the like. A job maybe identified by a job identifier that may be unique, for example, amongall the jobs for a particular customer. For example, the following HTTPPOST request may be used, in an illustrative embodiment, to initiate ajob to retrieve a data object identified by a data object identifier“dataObjectId.” In other embodiments, a data retrieval request mayrequest the retrieval of multiple data objects, data objects associatedwith a logical data container and the like.

POST /{accountId}/logical-data-container-name/data/{dataObjectId}HTTP/1.1

In response to the request, in an embodiment, archival data storagesystem 206 provides a job identifier job-id,” that is assigned to thejob in the following response. The response provides, in this example, apath to the storage location where the retrieved data will be stored.

HTTP/1.1 202 ACCEPTED Location:/{accountId}/logical-data-container-name/jobs/{job-id}

At any given point in time, the archival data storage system may havemany jobs pending for various data operations. In some embodiments, thearchival data storage system may employ job planning and optimizationtechniques such as batch processing, load balancing, job coalescence andthe like, to optimize system metrics such as cost, performance,scalability and the like. In some embodiments, the timing of the actualdata retrieval depends on factors such as the size of the retrieveddata, the system load and capacity, active status of storage devices andthe like. For example, in some embodiments, at least some data storagedevices in an archival data storage system may be activated orinactivated according to a power management schedule, for example, toreduce operational costs. Thus, retrieval of data stored in a currentlyactive storage device (such as a rotating hard drive) may be faster thanretrieval of data stored in a currently inactive storage device (such asa spinned-down hard drive).

In an embodiment, when a data retrieval job is completed, the retrieveddata is stored in a staging data store and made available for customerdownload. In some embodiments, a customer is notified of the change instatus of a job by a configurable notification service. In otherembodiments, a customer may learn of the status of a job by polling thesystem using a job identifier. The following HTTP GET request may beused, in an embodiment, to download data that is retrieved by a jobidentified by “job-id,” using a download path that has been previouslyprovided.

GET /{accountId}/logical-data-container-name/jobs/{job-id}/outputHTTP/1.1

In response to the GET request, in an illustrative embodiment, archivaldata storage system 206 may provide the retrieved data in the followingHTTP response, with a tree-hash of the data for verification purposes.

HTTP/1.1 200 OK Content-Length: 1128192 x-ABC-archive-description:“retrieved stuff” x-ABC-md5-tree-hash: 693d9a7838aff95c [1112192 bytesof user data follows]

In an embodiment, a customer may request the deletion of a data objectstored in an archival data storage system by specifying a data objectidentifier associated with the data object. For example, in anillustrative embodiment, a data object with data object identifier“dataObjectId” may be deleted using the following HTTP request. Inanother embodiment, a customer may request the deletion of multiple dataobjects such as those associated with a particular logical datacontainer.

DELETE /{accountId}/logical-data-container-name/data/{dataObjectId}HTTP/1.1

In various embodiments, data objects may be deleted in response to acustomer request or may be deleted automatically according to auser-specified or default expiration date. In some embodiments, dataobjects may be rendered inaccessible to customers upon an expirationtime but remain recoverable during a grace period beyond the expirationtime. In various embodiments, the grace period may be based onconfigurable information such as customer configuration, service-levelagreement terms and the like. In some embodiments, a customer may beprovided the abilities to query or receive notifications for pendingdata deletions and/or cancel one or more of the pending data deletions.For example, in one embodiment, a customer may set up notificationconfigurations associated with a logical data container such that thecustomer will receive notifications of certain events pertinent to thelogical data container. Such events may include the completion of a dataretrieval job request, the completion of metadata request, deletion ofdata objects or logical data containers and the like.

In an embodiment, archival data storage system 206 also providesmetadata APIs for retrieving and managing metadata such as metadataassociated with logical data containers. In various embodiments, suchrequests may be handled asynchronously (where results are returnedlater) or synchronously (where results are returned immediately).

Still referring to FIG. 2, in an embodiment, at least some of the APIrequests discussed above are handled by API request handler 218 as partof front end 208. For example, API request handler 218 may decode and/orparse an incoming API request to extract information, such as uniformresource identifier (“URI”), requested action and associated parameters,identity information, data object identifiers and the like. In addition,API request handler 218 invoke other services (described below), wherenecessary, to further process the API request.

In an embodiment, front end 208 includes an authentication service 220that may be invoked, for example, by API handler 218, to authenticate anAPI request. For example, in some embodiments, authentication service220 may verify identity information submitted with the API request suchas username and password Internet Protocol (“IP) address, cookies,digital certificate, digital signature and the like. In otherembodiments, authentication service 220 may require the customer toprovide additional information or perform additional steps toauthenticate the request, such as required in a multifactorauthentication scheme, under a challenge-response authenticationprotocol and the like.

In an embodiment, front end 208 includes an authorization service 222that may be invoked, for example, by API handler 218, to determinewhether a requested access is permitted according to one or morepolicies determined to be relevant to the request. For example, in oneembodiment, authorization service 222 verifies that a requested accessis directed to data objects contained in the requestor's own logicaldata containers or which the requester is otherwise authorized toaccess. In some embodiments, authorization service 222 or other servicesof front end 208 may check the validity and integrity of a data requestbased at least in part on information encoded in the request, such asvalidation information encoded by a data object identifier.

In an embodiment, front end 208 includes a metering service 224 thatmonitors service usage information for each customer such as datastorage space used, number of data objects stored, data requestsprocessed and the like. In an embodiment, front end 208 also includesaccounting service 226 that performs accounting and billing-relatedfunctionalities based, for example, on the metering informationcollected by the metering service 224, customer account information andthe like. For example, a customer may be charged a fee based on thestorage space used by the customer, size and number of the data objects,types and number of requests submitted, customer account type, servicelevel agreement the like.

In an embodiment, front end 208 batch processes some or all incomingrequests. For example, front end 208 may wait until a certain number ofrequests has been received before processing (e.g., authentication,authorization, accounting and the like) the requests. Such a batchprocessing of incoming requests may be used to gain efficiency.

In some embodiments, front end 208 may invoke services provided by othersubsystems of the archival data storage system to further process an APIrequest. For example, front end 208 may invoke services in metadataplane 216 to fulfill metadata requests. For another example, front end208 may stream data in and out of control plane for direct I/O 210 fordata storage and retrieval requests, respectively.

Referring now to control plane for direct I/O 210 illustrated in FIG. 2,in various embodiments, control plane for direct I/O 210 providesservices that create, track and manage jobs created as a result ofcustomer requests. As discussed above, a job refers to acustomer-initiated activity that may be performed asynchronously to theinitiating request, such as data retrieval, storage, metadata queries orthe like. In an embodiment, control plane for direct I/O 210 includes ajob tracker 230 that is configured to create job records or entriescorresponding to customer requests, such as those received from APIrequest handler 218, and monitor the execution of the jobs. In variousembodiments, a job record may include information related to theexecution of a job such as a customer account identifier, jobidentifier, data object identifier, reference to payload data cache 228(described below), job status, data validation information and the like.In some embodiments, job tracker 230 may collect information necessaryto construct a job record from multiple requests. For example, when alarge amount of data is requested to be stored, data upload may bebroken into multiple requests, each uploading a portion of the data. Insuch a case, job tracker 230 may maintain information to keep track ofthe upload status to ensure that all data parts have been receivedbefore a job record is created. In some embodiments, job tracker 230also obtains a data object identifier associated with the data to bestored and provides the data object identifier, for example, to a frontend service to be returned to a customer. In an embodiment, such dataobject identifier may be obtained from data plane 214 services such asstorage node manager 244, storage node registrar 248, and the like,described below.

In some embodiments, control plane for direct I/O 210 includes a jobtracker store 232 for storing job entries or records. In variousembodiments, job tracker store 230 may be implemented by a NoSQL datamanagement system, such as a key-value data store, a relational databasemanagement system (“RDBMS”) or any other data storage system. In someembodiments, data stored in job tracker store 230 may be partitioned toenable fast enumeration of jobs that belong to a specific customer,facilitate efficient bulk record deletion, parallel processing byseparate instances of a service and the like. For example, job trackerstore 230 may implement tables that are partitioned according tocustomer account identifiers and that use job identifiers as range keys.In an embodiment, job tracker store 230 is further sub-partitioned basedon time (such as job expiration time) to facilitate job expiration andcleanup operations. In an embodiment, transactions against job trackerstore 232 may be aggregated to reduce the total number of transactions.For example, in some embodiments, a job tracker 230 may performaggregate multiple jobs corresponding to multiple requests into onesingle aggregated job before inserting it into job tracker store 232.

In an embodiment, job tracker 230 is configured to submit the job forfurther job scheduling and planning, for example, by services in commoncontrol plane 212. Additionally, job tracker 230 may be configured tomonitor the execution of jobs and update corresponding job records injob tracker store 232 as jobs are completed. In some embodiments, jobtracker 230 may be further configured to handle customer queries such asjob status queries. In some embodiments, job tracker 230 also providesnotifications of job status changes to customers or other services ofthe archival data storage system. For example, when a data retrieval jobis completed, job tracker 230 may cause a customer to be notified (forexample, using a notification service) that data is available fordownload. As another example, when a data storage job is completed, jobtracker 230 may notify a cleanup agent 234 to remove payload dataassociated with the data storage job from a transient payload data cache228, described below.

In an embodiment, control plane for direct I/O 210 includes a payloaddata cache 228 for providing transient data storage services for payloaddata transiting between data plane 214 and front end 208. Such dataincludes incoming data pending storage and outgoing data pendingcustomer download. As used herein, transient data store is usedinterchangeably with temporary or staging data store to refer to a datastore that is used to store data objects before they are stored in anarchival data storage described herein or to store data objects that areretrieved from the archival data storage. A transient data store mayprovide volatile or non-volatile (durable) storage. In most embodiments,while potentially usable for persistently storing data, a transient datastore is intended to store data for a shorter period of time than anarchival data storage system and may be less cost-effective than thedata archival storage system described herein. In one embodiment,transient data storage services provided for incoming and outgoing datamay be differentiated. For example, data storage for the incoming data,which is not yet persisted in archival data storage, may provide higherreliability and durability than data storage for outgoing (retrieved)data, which is already persisted in archival data storage. In anotherembodiment, transient storage may be optional for incoming data, thatis, incoming data may be stored directly in archival data storagewithout being stored in transient data storage such as payload datacache 228, for example, when there is the system has sufficientbandwidth and/or capacity to do so.

In an embodiment, control plane for direct I/O 210 also includes acleanup agent 234 that monitors job tracker store 232 and/or payloaddata cache 228 and removes data that is no longer needed. For example,payload data associated with a data storage request may be safelyremoved from payload data cache 228 after the data is persisted inpermanent storage (e.g., data plane 214). On the reverse path, datastaged for customer download may be removed from payload data cache 228after a configurable period of time (e.g., 30 days since the data isstaged) or after a customer indicates that the staged data is no longerneeded.

In some embodiments, cleanup agent 234 removes a job record from jobtracker store 232 when the job status indicates that the job is completeor aborted. As discussed above, in some embodiments, job tracker store232 may be partitioned to enable to enable faster cleanup. In oneembodiment where data is partitioned by customer account identifiers,cleanup agent 234 may remove an entire table that stores jobs for aparticular customer account when the jobs are completed instead ofdeleting individual jobs one at a time. In another embodiment where datais further sub-partitioned based on job expiration time cleanup agent234 may bulk-delete a whole partition or table of jobs after all thejobs in the partition expire. In other embodiments, cleanup agent 234may receive instructions or control messages (such as indication thatjobs are completed) from other services such as job tracker 230 thatcause the cleanup agent 234 to remove job records from job tracker store232 and/or payload data cache 228.

Referring now to common control plane 212 illustrated in FIG. 2. Invarious embodiments, common control plane 212 provides a queue-basedload leveling service to dampen peak to average load levels (jobs)coming from control plane for I/O 210 and to deliver manageable workloadto data plane 214. In an embodiment, common control plane 212 includes ajob request queue 236 for receiving jobs created by job tracker 230 incontrol plane for direct I/O 210, described above, a storage nodemanager job store 240 from which services from data plane 214 (e.g.,storage node managers 244) pick up work to execute and a requestbalancer 238 for transferring job items from job request queue 236 tostorage node manager job store 240 in an intelligent manner.

In an embodiment, job request queue 236 provides a service for insertingitems into and removing items from a queue (e.g., first-in-first-out(FIFO) or first-in-last-out (FILO)), a set or any other suitable datastructure. Job entries in the job request queue 236 may be similar to ordifferent from job records stored in job tracker store 232, describedabove.

In an embodiment, common control plane 212 also provides a durable highefficiency job store, storage node manager job store 240, that allowsservices from data plane 214 (e.g., storage node manager 244,anti-entropy watcher 252) to perform job planning optimization, checkpointing and recovery. For example, in an embodiment, storage nodemanager job store 240 allows the job optimization such as batchprocessing, operation coalescing and the like by supporting scanning,querying, sorting or otherwise manipulating and managing job itemsstored in storage node manager job store 240. In an embodiment, astorage node manager 244 scans incoming jobs and sort the jobs by thetype of data operation (e.g., read, write or delete), storage locations(e.g., volume, disk), customer account identifier and the like. Thestorage node manager 244 may then reorder, coalesce, group in batches orotherwise manipulate and schedule the jobs for processing. For example,in one embodiment, the storage node manager 244 may batch process allthe write operations before all the read and delete operations. Inanother embodiment, the storage node manager 224 may perform operationcoalescing. For another example, the storage node manager 224 maycoalesce multiple retrieval jobs for the same object into one job orcancel a storage job and a deletion job for the same data object wherethe deletion job comes after the storage job.

In an embodiment, storage node manager job store 240 is partitioned, forexample, based on job identifiers, so as to allow independent processingof multiple storage node managers 244 and to provide even distributionof the incoming workload to all participating storage node managers 244.In various embodiments, storage node manager job store 240 may beimplemented by a NoSQL data management system, such as a key-value datastore, a RDBMS or any other data storage system.

In an embodiment, request balancer 238 provides a service fortransferring job items from job request queue 236 to storage nodemanager job store 240 so as to smooth out variation in workload and toincrease system availability. For example, request balancer 238 maytransfer job items from job request queue 236 at a lower rate or at asmaller granularity when there is a surge in job requests coming intothe job request queue 236 and vice versa when there is a lull inincoming job requests so as to maintain a relatively sustainable levelof workload in the storage node manager store 240. In some embodiments,such sustainable level of workload is around the same or below theaverage workload of the system.

In an embodiment, job items that are completed are removed from storagenode manager job store 240 and added to the job result queue 242. In anembodiment, data plane 214 services (e.g., storage node manager 244) areresponsible for removing the job items from the storage node manager jobstore 240 and adding them to job result queue 242. In some embodiments,job request queue 242 is implemented in a similar manner as job requestqueue 235, discussed above.

Referring now to data plane 214 illustrated in FIG. 2. In variousembodiments, data plane 214 provides services related to long-termarchival data storage, retrieval and deletion, data management andplacement, anti-entropy operations and the like. In various embodiments,data plane 214 may include any number and type of storage entities suchas data storage devices (such as tape drives, hard disk drives, solidstate devices, and the like), storage nodes or servers, datacenters andthe like. Such storage entities may be physical, virtual or anyabstraction thereof (e.g., instances of distributed storage and/orcomputing systems) and may be organized into any topology, includinghierarchical or tiered topologies. Similarly, the components of the dataplane may be dispersed, local or any combination thereof. For example,various computing or storage components may be local or remote to anynumber of datacenters, servers or data storage devices, which in turnmay be local or remote relative to one another. In various embodiments,physical storage entities may be designed for minimizing power andcooling costs by controlling the portions of physical hardware that areactive (e.g., the number of hard drives that are actively rotating). Inan embodiment, physical storage entities implement techniques, such asShingled Magnetic Recording (SMR), to increase storage capacity.

In an environment illustrated by FIG. 2, one or more storage nodemanagers 244 each controls one or more storage nodes 246 by sending andreceiving data and control messages. Each storage node 246 in turncontrols a (potentially large) collection of data storage devices suchas hard disk drives. In various embodiments, a storage node manager 244may communicate with one or more storage nodes 246 and a storage node246 may communicate with one or more storage node managers 244. In anembodiment, storage node managers 244 are implemented by one or morecomputing devices that are capable of performing relatively complexcomputations such as digest computation, data encoding and decoding, jobplanning and optimization and the like. In some embodiments, storagenodes 244 are implemented by one or more computing devices with lesspowerful computation capabilities than storage node managers 244.Further, in some embodiments the storage node manager 244 may not beincluded in the data path. For example, data may be transmitted from thepayload data cache 228 directly to the storage nodes 246 or from one ormore storage nodes 246 to the payload data cache 228. In this way, thestorage node manager 244 may transmit instructions to the payload datacache 228 and/or the storage nodes 246 without receiving the payloadsdirectly from the payload data cache 228 and/or storage nodes 246. Invarious embodiments, a storage node manager 244 may send instructions orcontrol messages to any other components of the archival data storagesystem 206 described herein to direct the flow of data.

In an embodiment, a storage node manager 244 serves as an entry pointfor jobs coming into and out of data plane 214 by picking job items fromcommon control plane 212 (e.g., storage node manager job store 240),retrieving staged data from payload data cache 228 and performingnecessary data encoding for data storage jobs and requesting appropriatestorage nodes 246 to store, retrieve or delete data. Once the storagenodes 246 finish performing the requested data operations, the storagenode manager 244 may perform additional processing, such as datadecoding and storing retrieved data in payload data cache 228 for dataretrieval jobs, and update job records in common control plane 212(e.g., removing finished jobs from storage node manager job store 240and adding them to job result queue 242).

In an embodiment, storage node manager 244 performs data encodingaccording to one or more data encoding schemes before data storage toprovide data redundancy, security and the like. Such data encodingschemes may include encryption schemes, redundancy encoding schemes suchas erasure encoding, redundant array of independent disks (RAID)encoding schemes, replication and the like. Likewise, in an embodiment,storage node managers 244 performs corresponding data decoding schemes,such as decryption, erasure-decoding and the like, after data retrievalto restore the original data.

As discussed above in connection with storage node manager job store240, storage node managers 244 may implement job planning andoptimizations such as batch processing, operation coalescing and thelike to increase efficiency. In some embodiments, jobs are partitionedamong storage node managers so that there is little or no overlapbetween the partitions. Such embodiments facilitate parallel processingby multiple storage node managers, for example, by reducing theprobability of racing or locking.

In various embodiments, data plane 214 is implemented to facilitate dataintegrity. For example, storage entities handling bulk data flows suchas storage nodes managers 244 and/or storage nodes 246 may validate thedigest of data stored or retrieved, check the error-detection code toensure integrity of metadata and the like.

In various embodiments, data plane 214 is implemented to facilitatescalability and reliability of the archival data storage system. Forexample, in one embodiment, storage node managers 244 maintain no orlittle internal state so that they can be added, removed or replacedwith little adverse impact. In one embodiment, each storage device is aself-contained and self-describing storage unit capable of providinginformation about data stored thereon. Such information may be used tofacilitate data recovery in case of data loss. Furthermore, in oneembodiment, each storage node 246 is capable of collecting and reportinginformation about the storage node including the network location of thestorage node and storage information of connected storage devices to oneor more storage node registrars 248 and/or storage node registrar stores250. In some embodiments, storage nodes 246 perform such self-reportingat system start up time and periodically provide updated information. Invarious embodiments, such a self-reporting approach provides dynamic andup-to-date directory information without the need to maintain a globalnamespace key map or index which can grow substantially as large amountsof data objects are stored in the archival data system.

In an embodiment, data plane 214 may also include one or more storagenode registrars 248 that provide directory information for storageentities and data stored thereon, data placement services and the like.Storage node registrars 248 may communicate with and act as a front endservice to one or more storage node registrar stores 250, which providestorage for the storage node registrars 248. In various embodiments,storage node registrar store 250 may be implemented by a NoSQL datamanagement system, such as a key-value data store, a RDBMS or any otherdata storage system. In some embodiments, storage node registrar stores250 may be partitioned to enable parallel processing by multipleinstances of services. As discussed above, in an embodiment, informationstored at storage node registrar store 250 is based at least partiallyon information reported by storage nodes 246 themselves.

In some embodiments, storage node registrars 248 provide directoryservice, for example, to storage node managers 244 that want todetermine which storage nodes 246 to contact for data storage, retrievaland deletion operations. For example, given a volume identifier providedby a storage node manager 244, storage node registrars 248 may provide,based on a mapping maintained in a storage node registrar store 250, alist of storage nodes that host volume components corresponding to thevolume identifier. Specifically, in one embodiment, storage noderegistrar store 250 stores a mapping between a list of identifiers ofvolumes or volume components and endpoints, such as Domain Name System(DNS) names, of storage nodes that host the volumes or volumecomponents.

As used herein, a “volume” refers to a logical storage space within adata storage system in which data objects may be stored. A volume may beidentified by a volume identifier. A volume may reside in one physicalstorage device (e.g., a hard disk) or span across multiple storagedevices. In the latter case, a volume comprises a plurality of volumecomponents each residing on a different storage device. As used herein,a “volume component” refers a portion of a volume that is physicallystored in a storage entity such as a storage device. Volume componentsfor the same volume may be stored on different storage entities. In oneembodiment, when data is encoded by a redundancy encoding scheme (e.g.,erasure coding scheme, RAID, replication), each encoded data componentor “shard” may be stored in a different volume component to providefault tolerance and isolation. In some embodiments, a volume componentis identified by a volume component identifier that includes a volumeidentifier and a shard slot identifier. As used herein, a shard slotidentifies a particular shard, row or stripe of data in a redundancyencoding scheme. For example, in one embodiment, a shard slotcorresponds to an erasure coding matrix row. In some embodiments,storage node registrar store 250 also stores information about volumesor volume components such as total, used and free space, number of dataobjects stored and the like.

In some embodiments, data plane 214 also includes a storage allocator256 for allocating storage space (e.g., volumes) on storage nodes tostore new data objects, based at least in part on information maintainedby storage node registrar store 250, to satisfy data isolation and faulttolerance constraints. In some embodiments, storage allocator 256requires manual intervention.

In some embodiments, data plane 214 also includes an anti-entropywatcher 252 for detecting entropic effects and initiating anti-entropycorrection routines. For example, anti-entropy watcher 252 may beresponsible for monitoring activities and status of all storage entitiessuch as storage nodes, reconciling live or actual data with maintaineddata and the like. In various embodiments, entropic effects include, butare not limited to, performance degradation due to data fragmentationresulting from repeated write and rewrite cycles, hardware wear (e.g.,of magnetic media), data unavailability and/or data loss due tohardware/software malfunction, environmental factors, physicaldestruction of hardware, random chance or other causes. Anti-entropywatcher 252 may detect such effects and in some embodiments maypreemptively and/or reactively institute anti-entropy correctionroutines and/or policies.

In an embodiment, anti-entropy watcher 252 causes storage nodes 246 toperform periodic anti-entropy scans on storage devices connected to thestorage nodes. Anti-entropy watcher 252 may also inject requests in jobrequest queue 236 (and subsequently job result queue 242) to collectinformation, recover data and the like. In some embodiments,anti-entropy watcher 252 may perform scans, for example, on cold indexstore 262, described below, and storage nodes 246, to ensure referentialintegrity.

In an embodiment, information stored at storage node registrar store 250is used by a variety of services such as storage node registrar 248,storage allocator 256, anti-entropy watcher 252 and the like. Forexample, storage node registrar 248 may provide data location andplacement services (e.g., to storage node managers 244) during datastorage, retrieval and deletion. For example, given the size of a dataobject to be stored and information maintained by storage node registrarstore 250, a storage node registrar 248 may determine where (e.g.,volume) to store the data object and provides an indication of thestorage location of the data object which may be used to generate a dataobject identifier associated with the data object. As another example,in an embodiment, storage allocator 256 uses information stored instorage node registrar store 250 to create and place volume componentsfor new volumes in specific storage nodes to satisfy isolation and faulttolerance constraints. As yet another example, in an embodiment,anti-entropy watcher 252 uses information stored in storage noderegistrar store 250 to detect entropic effects such as data loss,hardware failure and the like.

In some embodiments, data plane 214 also includes an orphan cleanup datastore 254, which is used to track orphans in the storage system. As usedherein, an orphan is a stored data object that is not referenced by anyexternal entity. In various embodiments, orphan cleanup data store 254may be implemented by a NoSQL data management system, such as akey-value data store, an RDBMS or any other data storage system. In someembodiments, storage node registrars 248 stores object placementinformation in orphan cleanup data store 254. Subsequently, informationstored in orphan cleanup data store 254 may be compared, for example, byan anti-entropy watcher 252, with information maintained in metadataplane 216. If an orphan is detected, in some embodiments, a request isinserted in the common control plane 212 to delete the orphan.

Referring now to metadata plane 216 illustrated in FIG. 2. In variousembodiments, metadata plane 216 provides information about data objectsstored in the system for inventory and accounting purposes, to satisfycustomer metadata inquiries and the like. In the illustrated embodiment,metadata plane 216 includes a metadata manager job store 258 whichstores information about executed transactions based on entries from jobresult queue 242 in common control plane 212. In various embodiments,metadata manager job store 258 may be implemented by a NoSQL datamanagement system, such as a key-value data store, a RDBMS or any otherdata storage system. In some embodiments, metadata manager job store 258is partitioned and sub-partitioned, for example, based on logical datacontainers, to facilitate parallel processing by multiple instances ofservices such as metadata manager 260.

In the illustrative embodiment, metadata plane 216 also includes one ormore metadata managers 260 for generating a cold index of data objects(e.g., stored in cold index store 262) based on records in metadatamanager job store 258. As used herein, a “cold” index refers to an indexthat is updated infrequently. In various embodiments, a cold index ismaintained to reduce cost overhead. In some embodiments, multiplemetadata managers 260 may periodically read and process records fromdifferent partitions in metadata manager job store 258 in parallel andstore the result in a cold index store 262.

In some embodiments cold index store 262 may be implemented by areliable and durable data storage service. In some embodiments, coldindex store 262 is configured to handle metadata requests initiated bycustomers. For example, a customer may issue a request to list all dataobjects contained in a given logical data container. In response to sucha request, cold index store 262 may provide a list of identifiers of alldata objects contained in the logical data container based oninformation maintained by cold index 262. In some embodiments, anoperation may take a relative long period of time and the customer maybe provided a job identifier to retrieve the result when the job isdone. In other embodiments, cold index store 262 is configured to handleinquiries from other services, for example, from front end 208 forinventory, accounting and billing purposes.

In some embodiments, metadata plane 216 may also include a containermetadata store 264 that stores information about logical data containerssuch as container ownership, policies, usage and the like. Suchinformation may be used, for example, by front end 208 services, toperform authorization, metering, accounting and the like. In variousembodiments, container metadata store 264 may be implemented by a NoSQLdata management system, such as a key-value data store, a RDBMS or anyother data storage system.

As described herein, in various embodiments, the archival data storagesystem 206 described herein is implemented to be efficient and scalable.For example, in an embodiment, batch processing and request coalescingis used at various stages (e.g., front end request handling, controlplane job request handling, data plane data request handling) to improveefficiency. For another example, in an embodiment, processing ofmetadata such as jobs, requests and the like are partitioned so as tofacilitate parallel processing of the partitions by multiple instancesof services.

In an embodiment, data elements stored in the archival data storagesystem (such as data components, volumes, described below) areself-describing so as to avoid the need for a global index datastructure. For example, in an embodiment, data objects stored in thesystem may be addressable by data object identifiers that encode storagelocation information. For another example, in an embodiment, volumes maystore information about which data objects are stored in the volume andstorage nodes and devices storing such volumes may collectively reporttheir inventory and hardware information to provide a global view of thedata stored in the system (such as evidenced by information stored instorage node registrar store 250). In such an embodiment, the globalview is provided for efficiency only and not required to locate datastored in the system.

In various embodiments, the archival data storage system describedherein is implemented to improve data reliability and durability. Forexample, in an embodiment, a data object is redundantly encoded into aplurality of data components and stored across different data storageentities to provide fault tolerance. For another example, in anembodiment, data elements have multiple levels of integrity checks. Inan embodiment, parent/child relations always have additional informationto ensure full referential integrity. For example, in an embodiment,bulk data transmission and storage paths are protected by having theinitiator pre-calculate the digest on the data before transmission andsubsequently supply the digest with the data to a receiver. The receiverof the data transmission is responsible for recalculation, comparing andthen acknowledging to the sender that includes the recalculated thedigest. Such data integrity checks may be implemented, for example, byfront end services, transient data storage services, data plane storageentities and the like described above.

FIG. 3 illustrates an interconnection network 300 in which components ofan archival data storage system may be connected, in accordance with atleast one embodiment. In particular, the illustrated example shows howdata plane components are connected to the interconnection network 300.In some embodiments, the interconnection network 300 may include a fattree interconnection network where the link bandwidth grows higher or“fatter” towards the root of the tree. In the illustrated example, dataplane includes one or more datacenters 301. Each datacenter 301 mayinclude one or more storage node manager server racks 302 where eachserver rack hosts one or more servers that collectively provide thefunctionality of a storage node manager such as described in connectionwith FIG. 2. In other embodiments, each storage node manager server rackmay host more than one storage node manager. Configuration parameterssuch as number of storage node managers per rack, number of storage nodemanager racks and the like may be determined based on factors such ascost, scalability, redundancy and performance requirements, hardware andsoftware resources and the like.

Each storage node manager server rack 302 may have a storage nodemanager rack connection 314 to an interconnect 308 used to connect tothe interconnection network 300. In some embodiments, the connection 314is implemented using a network switch 303 that may include a top-of-rackEthernet switch or any other type of network switch. In variousembodiments, interconnect 308 is used to enable high-bandwidth andlow-latency bulk data transfers. For example, interconnect may include aClos network, a fat tree interconnect, an Asynchronous Transfer Mode(ATM) network, a Fast or Gigabit Ethernet and the like.

In various embodiments, the bandwidth of storage node manager rackconnection 314 may be configured to enable high-bandwidth andlow-latency communications between storage node managers and storagenodes located within the same or different data centers. For example, inan embodiment, the storage node manager rack connection 314 has abandwidth of 10 Gigabit per second (Gbps).

In some embodiments, each datacenter 301 may also include one or morestorage node server racks 304 where each server rack hosts one or moreservers that collectively provide the functionalities of a number ofstorage nodes such as described in connection with FIG. 2. Configurationparameters such as number of storage nodes per rack, number of storagenode racks, ration between storage node managers and storage nodes andthe like may be determined based on factors such as cost, scalability,redundancy and performance requirements, hardware and software resourcesand the like. For example, in one embodiment, there are 3 storage nodesper storage node server rack, 30-80 racks per data center and a storagenodes/storage node manager ratio of 10 to 1.

Each storage node server rack 304 may have a storage node rackconnection 316 to an interconnection network switch 308 used to connectto the interconnection network 300. In some embodiments, the connection316 is implemented using a network switch 305 that may include atop-of-rack Ethernet switch or any other type of network switch. Invarious embodiments, the bandwidth of storage node rack connection 316may be configured to enable high-bandwidth and low-latencycommunications between storage node managers and storage nodes locatedwithin the same or different data centers. In some embodiments, astorage node rack connection 316 has a higher bandwidth than a storagenode manager rack connection 314. For example, in an embodiment, thestorage node rack connection 316 has a bandwidth of 20 Gbps while astorage node manager rack connection 314 has a bandwidth of 10 Gbps.

In some embodiments, datacenters 301 (including storage node managersand storage nodes) communicate, via connection 310, with other computingresources services 306 such as payload data cache 228, storage nodemanager job store 240, storage node registrar 248, storage noderegistrar store 350, orphan cleanup data store 254, metadata manager jobstore 258 and the like as described in connection with FIG. 2.

In some embodiments, one or more datacenters 301 may be connected viainter-datacenter connection 312. In some embodiments, connections 310and 312 may be configured to achieve effective operations and use ofhardware resources. For example, in an embodiment, connection 310 has abandwidth of 30-100 Gbps per datacenter and inter-datacenter connection312 has a bandwidth of 100-250 Gbps.

FIG. 4 illustrates an interconnection network 400 in which components ofan archival data storage system may be connected, in accordance with atleast one embodiment. In particular, the illustrated example shows hownon-data plane components are connected to the interconnection network300. As illustrated, front end services, such as described in connectionwith FIG. 2, may be hosted by one or more front end server racks 402.For example, each front end server rack 402 may host one or more webservers. The front end server racks 402 may be connected to theinterconnection network 400 via a network switch 408. In one embodiment,configuration parameters such as number of front end services, number ofservices per rack, bandwidth for front end server rack connection 314and the like may roughly correspond to those for storage node managersas described in connection with FIG. 3.

In some embodiments, control plane services and metadata plane servicesas described in connection with FIG. 2 may be hosted by one or moreserver racks 404. Such services may include job tracker 230, metadatamanager 260, cleanup agent 232, job request balancer 238 and otherservices. In some embodiments, such services include services that donot handle frequent bulk data transfers. Finally, components describedherein may communicate via connection 410, with other computingresources services 406 such as payload data cache 228, job tracker store232, metadata manager job store 258 and the like as described inconnection with FIG. 2.

FIG. 5 illustrates an example process 500 for storing data, inaccordance with at least one embodiment. Some or all of process 500 (orany other processes described herein or variations and/or combinationsthereof) may be performed under the control of one or more computersystems configured with executable instructions and may be implementedas code (e.g., executable instructions, one or more computer programs orone or more applications) executing collectively on one or moreprocessors, by hardware or combinations thereof. The code may be storedon a computer-readable storage medium, for example, in the form of acomputer program comprising a plurality of instructions executable byone or more processors. The computer-readable storage medium may benon-transitory. In an embodiment, one or more components of archivaldata storage system 206 as described in connection with FIG. 2 mayperform process 500.

In an embodiment, process 500 includes receiving 502 a data storagerequest to store archival data such as a document, a video or audio fileor the like. Such a data storage request may include payload data andmetadata such as size and digest of the payload data, useridentification information (e.g., user name, account identifier and thelike), a logical data container identifier and the like. In someembodiments, process 500 may include receiving 502 multiple storagerequests each including a portion of larger payload data. In otherembodiments, a storage request may include multiple data objects to beuploaded. In an embodiment, step 502 of process 500 is implemented by aservice such as API request handler 218 of front end 208 as described inconnection with FIG. 2.

In an embodiment, process 500 includes processing 504 the storagerequest upon receiving 502 the request. Such processing may include, forexample, verifying the integrity of data received, authenticating thecustomer, authorizing requested access against access control policies,performing meter- and accounting-related activities and the like. In anembodiment, such processing may be performed by services of front end208 such as described in connection with FIG. 2. In an embodiment, sucha request may be processed in connection with other requests, forexample, in batch mode.

In an embodiment, process 500 includes storing 506 the data associatedwith the storage request in a staging data store. Such staging datastore may include a transient data store such as provided by payloaddata cache 228 as described in connection with FIG. 2. In someembodiments, only payload data is stored in the staging store. In otherembodiments, metadata related to the payload data may also be stored inthe staging store. In an embodiment, data integrity is validated (e.g.,based on a digest) before being stored at a staging data store.

In an embodiment, process 500 includes providing 508 a data objectidentifier associated with the data to be stored, for example, in aresponse to the storage request. As described above, a data objectidentifier may be used by subsequent requests to retrieve, delete orotherwise reference data stored. In an embodiment, a data objectidentifier may encode storage location information that may be used tolocate the stored data object, payload validation information such assize, digest, timestamp and the like that may be used to validate theintegrity of the payload data, metadata validation information such aserror-detection codes that may be used to validate the integrity ofmetadata such as the data object identifier itself and informationencoded in the data object identifier and the like. In an embodiment, adata object identifier may also encode information used to validate orauthorize subsequent customer requests. For example, a data objectidentifier may encode the identifier of the logical data container thatthe data object is stored in. In a subsequent request to retrieve thisdata object, the logical data container identifier may be used todetermine whether the requesting entity has access to the logical datacontainer and hence the data objects contained therein. In someembodiments, the data object identifier may encode information based oninformation supplied by a customer (e.g., a global unique identifier,GUID, for the data object and the like) and/or information collected orcalculated by the system performing process 500 (e.g., storage locationinformation). In some embodiments, generating a data object identifiermay include encrypting some or all of the information described aboveusing a cryptographic private key. In some embodiments, thecryptographic private key may be periodically rotated. In someembodiments, a data object identifier may be generated and/or providedat a different time than described above. For example, a data objectidentifier may be generated and/or provided after a storage job(described below) is created and/or completed.

In an embodiment, providing 508 a data object identifier may includedetermining a storage location for the before the data is actuallystored there. For example, such determination may be based at least inpart on inventory information about existing data storage entities suchas operational status (e.g., active or inactive), available storagespace, data isolation requirement and the like. In an environment suchas environment 200 illustrated by FIG. 2, such determination may beimplemented by a service such as storage node registrar 248 as describedabove in connection with FIG. 2. In some embodiments, such determinationmay include allocating new storage space (e.g., volume) on one or morephysical storage devices by a service such as storage allocator 256 asdescribed in connection with FIG. 2.

In an embodiment, a storage location identifier may be generated torepresent the storage location determined above. Such a storage locationidentifier may include, for example, a volume reference object whichcomprises a volume identifier component and data object identifiercomponent. The volume reference component may identify the volume thedata is stored on and the data object identifier component may identifywhere in the volume the data is stored. In general, the storage locationidentifier may comprise components that identify various levels within alogical or physical data storage topology (such as a hierarchy) in whichdata is organized. In some embodiments, the storage location identifiermay point to where actual payload data is stored or a chain of referenceto where the data is stored.

In an embodiments, a data object identifier encodes a digest (e.g., ahash) of at least a portion of the data to be stored, such as thepayload data. In some embodiments, the digest may be based at least inpart on a customer-provided digest. In other embodiments, the digest maybe calculated from scratch based on the payload data.

In an embodiment, process 500 includes creating 510 a storage job forpersisting data to a long-term data store and scheduling 512 the storagejob for execution. In environment 200 as described in connection withFIG. 2, steps 508, 510 and 512 may be implemented at least in part bycomponents of control plane for direct I/O 210 and common control plane212 as described above. Specifically, in an embodiment, job tracker 230creates a job record and stores the job record in job tracker store 232.As described above, job tracker 230 may perform batch processing toreduce the total number of transactions against job tracker store 232.Additionally, job tracker store 232 may be partitioned or otherwiseoptimized to facilitate parallel processing, cleanup operations and thelike. A job record, as described above, may include job-relatedinformation such as a customer account identifier, job identifier,storage location identifier, reference to data stored in payload datacache 228, job status, job creation and/or expiration time and the like.In some embodiments, a storage job may be created before a data objectidentifier is generated and/or provided. For example, a storage jobidentifier, instead of or in addition to a data object identifier, maybe provided in response to a storage request at step 508 above.

In an embodiment, scheduling 512 the storage job for execution includesperforming job planning and optimization, such as queue-based loadleveling or balancing, job partitioning and the like, as described inconnection with common control plane 212 of FIG. 2. For example, in anembodiment, job request balancer 238 transfers job items from jobrequest queue 236 to storage node manager job store 240 according to ascheduling algorithm so as to dampen peak to average load levels (jobs)coming from control plane for I/O 210 and to deliver manageable workloadto data plane 214. As another example, storage node manager job store240 may be partitioned to facilitate parallel processing of the jobs bymultiple workers such as storage node managers 244. As yet anotherexample, storage node manager job store 240 may provide querying,sorting and other functionalities to facilitate batch processing andother job optimizations.

In an embodiment, process 500 includes selecting 514 the storage job forexecution, for example, by a storage node manager 244 from storage nodemanager job stored 240 as described in connection with FIG. 2. Thestorage job may be selected 514 with other jobs for batch processing orotherwise selected as a result of job planning and optimizationdescribed above.

In an embodiment, process 500 includes obtaining 516 data from a stagingstore, such as payload data cache 228 described above in connection withFIG. 2. In some embodiments, the integrity of the data may be checked,for example, by verifying the size, digest, an error-detection code andthe like.

In an embodiment, process 500 includes obtaining 518 one or more dataencoding schemes such as an encryption scheme, a redundancy encodingscheme such as erasure encoding, redundant array of independent disks(RAID) encoding schemes, replication, and the like. In some embodiments,such encoding schemes evolve to adapt to different requirements. Forexample, encryption keys may be rotated periodically and stretch factorof an erasure coding scheme may be adjusted over time to differenthardware configurations, redundancy requirements and the like.

In an embodiment, process 500 includes encoding 520 with the obtainedencoding schemes. For example, in an embodiment, data is encrypted andthe encrypted data is erasure-encoded. In an embodiment, storage nodemanagers 244 described in connection with FIG. 2 may be configured toperform the data encoding described herein. In an embodiment,application of such encoding schemes generates a plurality of encodeddata components or shards, which may be stored across different storageentities such as storage devices, storage nodes, datacenters and thelike to provide fault tolerance. In an embodiment where data maycomprise multiple parts (such as in the case of a multi-part upload),each part may be encoded and stored as described herein.

In an embodiment, process 500 includes determining 522 the storageentities for such encoded data components. For example, in anenvironment 200 illustrated by FIG. 2, a storage node manager 244 maydetermine the plurality of storage nodes 246 to store the encoded datacomponents by querying a storage node registrar 248 using a volumeidentifier. Such a volume identifier may be part of a storage locationidentifier associated with the data to be stored. In response to thequery with a given volume identifier, in an embodiment, storage noderegistrar 248 returns a list of network locations (including endpoints,DNS names, IP addresses and the like) of storage nodes 246 to store theencoded data components. As described in connection with FIG. 2, storagenode registrar 248 may determine such a list based on self-reported anddynamically provided and/or updated inventory information from storagenodes 246 themselves. In some embodiments, such determination is basedon data isolation, fault tolerance, load balancing, power conservation,data locality and other considerations. In some embodiments, storageregistrar 248 may cause new storage space to be allocated, for example,by invoking storage allocator 256 as described in connection with FIG.2.

In an embodiment, process 500 includes causing 524 storage of theencoded data component(s) at the determined storage entities. Forexample, in an environment 200 illustrated by FIG. 2, a storage nodemanager 244 may request each of the storage nodes 246 determined aboveto store a data component at a given storage location. Each of thestorage nodes 246, upon receiving the storage request from storage nodemanager 244 to store a data component, may cause the data component tobe stored in a connected storage device. In some embodiments, at least aportion of the data object identifier is stored with all or some of thedata components in either encoded or unencoded form. For example, thedata object identifier may be stored in the header of each datacomponent and/or in a volume component index stored in a volumecomponent. In some embodiments, a storage node 246 may perform batchprocessing or other optimizations to process requests from storage nodemanagers 244.

In an embodiment, a storage node 246 sends an acknowledgement to therequesting storage node manager 244 indicating whether data is storedsuccessfully. In some embodiments, a storage node 246 returns an errormessage, when for some reason, the request cannot be fulfilled. Forexample, if a storage node receives two requests to store to the samestorage location, one or both requests may fail. In an embodiment, astorage node 246 performs validation checks prior to storing the dataand returns an error if the validation checks fail. For example, dataintegrity may be verified by checking an error-detection code or adigest. As another example, storage node 246 may verify, for example,based on a volume index, that the volume identified by a storage requestis stored by the storage node and/or that the volume has sufficientspace to store the data component.

In some embodiments, data storage is considered successful when storagenode manager 244 receives positive acknowledgement from at least asubset (a storage quorum) of requested storage nodes 246. In someembodiments, a storage node manager 244 may wait until the receipt of aquorum of acknowledgement before removing the state necessary to retrythe job. Such state information may include encoded data components forwhich an acknowledgement has not been received. In other embodiments, toimprove the throughput, a storage node manager 244 may remove the statenecessary to retry the job before receiving a quorum of acknowledgement

In an embodiment, process 500 includes updating 526 metadata informationincluding, for example, metadata maintained by data plane 214 (such asindex and storage space information for a storage device, mappinginformation stored at storage node registrar store 250 and the like),metadata maintained by control planes 210 and 212 (such as job-relatedinformation), metadata maintained by metadata plane 216 (such as a coldindex) and the like. In various embodiments, some of such metadatainformation may be updated via batch processing and/or on a periodicbasis to reduce performance and cost impact. For example, in data plane214, information maintained by storage node registrar store 250 may beupdated to provide additional mapping of the volume identifier of thenewly stored data and the storage nodes 246 on which the data componentsare stored, if such a mapping is not already there. For another example,volume index on storage devices may be updated to reflect newly addeddata components.

In common control plane 212, job entries for completed jobs may beremoved from storage node manager job store 240 and added to job resultqueue 242 as described in connection with FIG. 2. In control plane fordirect I/O 210, statuses of job records in job tracker store 232 may beupdated, for example, by job tracker 230 which monitors the job resultqueue 242. In various embodiments, a job that fails to complete may beretried for a number of times. For example, in an embodiment, a new jobmay be created to store the data at a different location. As anotherexample, an existing job record (e.g., in storage node manager job store240, job tracker store 232 and the like) may be updated to facilitateretry of the same job.

In metadata plane 216, metadata may be updated to reflect the newlystored data. For example, completed jobs may be pulled from job resultqueue 242 into metadata manager job store 258 and batch-processed bymetadata manager 260 to generate an updated index such as stored in coldindex store 262. For another example, customer information may beupdated to reflect changes for metering and accounting purposes.

Finally, in some embodiments, once a storage job is completedsuccessfully, job records, payload data and other data associated with astorage job may be removed, for example, by a cleanup agent 234 asdescribed in connection with FIG. 2. In some embodiments, such removalmay be processed by batch processing, parallel processing or the like.

FIG. 6 illustrates an example process 500 for retrieving data, inaccordance with at least one embodiment. In an embodiment, one or morecomponents of archival data storage system 206 as described inconnection with FIG. 2 collectively perform process 600.

In an embodiment, process 600 includes receiving 602 a data retrievalrequest to retrieve data such as stored by process 500, described above.Such a data retrieval request may include a data object identifier, suchas provided by step 508 of process 500, described above, or any otherinformation that may be used to identify the data to be retrieved.

In an embodiment, process 600 includes processing 604 the data retrievalrequest upon receiving 602 the request. Such processing may include, forexample, authenticating the customer, authorizing requested accessagainst access control policies, performing meter and accounting relatedactivities and the like. In an embodiment, such processing may beperformed by services of front end 208 such as described in connectionwith FIG. 2. In an embodiment, such request may be processed inconnection with other requests, for example, in batch mode.

In an embodiment, processing 604 the retrieval request may be based atleast in part on the data object identifier that is included in theretrieval request. As described above, data object identifier may encodestorage location information, payload validation information such assize, creation timestamp, payload digest and the like, metadatavalidation information, policy information and the like. In anembodiment, processing 604 the retrieval request includes decoding theinformation encoded in the data object identifier, for example, using aprivate cryptographic key and using at least some of the decodedinformation to validate the retrieval request. For example, policyinformation may include access control information that may be used tovalidate that the requesting entity of the retrieval request has therequired permission to perform the requested access. As another example,metadata validation information may include an error-detection code suchas a cyclic redundancy check (“CRC”) that may be used to verify theintegrity of data object identifier or a component of it.

In an embodiment, process 600 includes creating 606 a data retrieval jobcorresponding to the data retrieval request and providing 608 a jobidentifier associated with the data retrieval job, for example, in aresponse to the data retrieval request. In some embodiments, creating606 a data retrieval job is similar to creating a data storage job asdescribed in connection with step 510 of process 500 illustrated in FIG.5. For example, in an embodiment, a job tracker 230 may create a jobrecord that includes at least some information encoded in the dataobject identifier and/or additional information such as a job expirationtime and the like and store the job record in job tracker store 232. Asdescribed above, job tracker 230 may perform batch processing to reducethe total number of transactions against job tracker store 232.Additionally, job tracker store 232 may be partitioned or otherwiseoptimized to facilitate parallel processing, cleanup operations and thelike.

In an embodiment, process 600 includes scheduling 610 the data retrievaljob created above. In some embodiments, scheduling 610 the dataretrieval job for execution includes performing job planning andoptimization such as described in connection with step 512 of process500 of FIG. 5. For example, the data retrieval job may be submitted intoa job queue and scheduled for batch processing with other jobs based atleast in part on costs, power management schedules and the like. Foranother example, the data retrieval job may be coalesced with otherretrieval jobs based on data locality and the like.

In an embodiment, process 600 includes selecting 612 the data retrievaljob for execution, for example, by a storage node manager 244 fromstorage node manager job stored 240 as described in connection with FIG.2. The retrieval job may be selected 612 with other jobs for batchprocessing or otherwise selected as a result of job planning andoptimization described above.

In an embodiment, process 600 includes determining 614 the storageentities that store the encoded data components that are generated by astorage process such as process 500 described above. In an embodiment, astorage node manager 244 may determine a plurality of storage nodes 246to retrieve the encoded data components in a manner similar to thatdiscussed in connection with step 522 of process 500, above. Forexample, such determination may be based on load balancing, powerconservation, efficiency and other considerations.

In an embodiment, process 600 includes determining 616 one or more datadecoding schemes that may be used to decode retrieved data. Typically,such decoding schemes correspond to the encoding schemes applied to theoriginal data when the original data is previously stored. For example,such decoding schemes may include decryption with a cryptographic key,erasure-decoding and the like.

In an embodiment, process 600 includes causing 618 retrieval of at leastsome of the encoded data components from the storage entities determinedin step 614 of process 600. For example, in an environment 200illustrated by FIG. 2, a storage node manager 244 responsible for thedata retrieval job may request a subset of storage nodes 246 determinedabove to retrieve their corresponding data components. In someembodiments, a minimum number of encoded data components is needed toreconstruct the original data where the number may be determined basedat least in part on the data redundancy scheme used to encode the data(e.g., stretch factor of an erasure coding). In such embodiments, thesubset of storage nodes may be selected such that no less than theminimum number of encoded data components is retrieved.

Each of the subset of storage nodes 246, upon receiving a request fromstorage node manager 244 to retrieve a data component, may validate therequest, for example, by checking the integrity of a storage locationidentifier (that is part of the data object identifier), verifying thatthe storage node indeed holds the requested data component and the like.Upon a successful validation, the storage node may locate the datacomponent based at least in part on the storage location identifier. Forexample, as described above, the storage location identifier may includea volume reference object which comprises a volume identifier componentand a data object identifier component where the volume referencecomponent to identify the volume the data is stored and a data objectidentifier component may identify where in the volume the data isstored. In an embodiment, the storage node reads the data component, forexample, from a connected data storage device and sends the retrieveddata component to the storage node manager that requested the retrieval.In some embodiments, the data integrity is checked, for example, byverifying the data component identifier or a portion thereof isidentical to that indicated by the data component identifier associatedwith the retrieval job. In some embodiments, a storage node may performbatching or other job optimization in connection with retrieval of adata component.

In an embodiment, process 600 includes decoding 620, at least theminimum number of the retrieved encoded data components with the one ormore data decoding schemes determined at step 616 of process 600. Forexample, in one embodiment, the retrieved data components may be erasuredecoded and then decrypted. In some embodiments, a data integrity checkis performed on the reconstructed data, for example, using payloadintegrity validation information encoded in the data object identifier(e.g., size, timestamp, digest). In some cases, the retrieval job mayfail due to a less-than-minimum number of retrieved data components,failure of data integrity check and the like. In such cases, theretrieval job may be retried in a fashion similar to that described inconnection with FIG. 5. In some embodiments, the original data comprisesmultiple parts of data and each part is encoded and stored. In suchembodiments, during retrieval, the encoded data components for each partof the data may be retrieved and decoded (e.g., erasure-decoded anddecrypted) to form the original part and the decoded parts may becombined to form the original data.

In an embodiment, process 600 includes storing reconstructed data in astaging store such as payload data cache 228 described in connectionwith FIG. 2. In some embodiments, data stored 622 in the staging storemay be available for download by a customer for a period of time orindefinitely. In an embodiment, data integrity may be checked (e.g.,using a digest) before the data is stored in the staging store.

In an embodiment, process 600 includes providing 624 a notification ofthe completion of the retrieval job to the requestor of the retrievalrequest or another entity or entities otherwise configured to receivesuch a notification. Such notifications may be provided individually orin batches. In other embodiments, the status of the retrieval job may beprovided upon a polling request, for example, from a customer.

FIG. 7 illustrates an example process 700 for deleting data, inaccordance with at least one embodiment. In an embodiment, one or morecomponents of archival data storage system 206 as described inconnection with FIG. 2 collectively perform process 700.

In an embodiment, process 700 includes receiving 702 a data deletionrequest to delete data such as stored by process 500, described above.Such a data retrieval request may include a data object identifier, suchas provided by step 508 of process 500, described above, or any otherinformation that may be used to identify the data to be deleted.

In an embodiment, process 700 includes processing 704 the data deletionrequest upon receiving 702 the request. In some embodiments, theprocessing 704 is similar to that for step 504 of process 500 and step604 of process 600, described above. For example, in an embodiment, theprocessing 704 is based at least in part on the data object identifierthat is included in the data deletion request.

In an embodiment, process 700 includes creating 706 a data retrieval jobcorresponding to the data deletion request. Such a retrieval job may becreated similar to the creation of storage job described in connectionwith step 510 of process 500 and the creation of the retrieval jobdescribed in connection with step 606 of process 600.

In an embodiment, process 700 includes providing 708 an acknowledgementthat the data is deleted. In some embodiments, such acknowledgement maybe provided in response to the data deletion request so as to provide aperception that the data deletion request is handled synchronously. Inother embodiments, a job identifier associated with the data deletionjob may be provided similar to the providing of job identifiers for dataretrieval requests.

In an embodiment, process 700 includes scheduling 708 the data deletionjob for execution. In some embodiments, scheduling 708 of data deletionjobs may be implemented similar to that described in connection withstep 512 of process 500 and in connection with step 610 of process 600,described above. For example, data deletion jobs for closely-locateddata may be coalesced and/or batch processed. For another example, datadeletion jobs may be assigned a lower priority than data retrieval jobs.

In some embodiments, data stored may have an associated expiration timethat is specified by a customer or set by default. In such embodiments,a deletion job may be created 706 and schedule 710 automatically on ornear the expiration time of the data. In some embodiments, theexpiration time may be further associated with a grace period duringwhich data is still available or recoverable. In some embodiments, anotification of the pending deletion may be provided before, on or afterthe expiration time.

In some embodiments, process 700 includes selecting 712 the datadeletion job for execution, for example, by a storage node manager 244from storage node manager job stored 240 as described in connection withFIG. 2. The deletion job may be selected 712 with other jobs for batchprocessing or otherwise selected as a result of job planning andoptimization described above.

In some embodiments, process 700 includes determining 714 the storageentities for data components that store the data components that aregenerated by a storage process such as process 500 described above. Inan embodiment, a storage node manager 244 may determine a plurality ofstorage nodes 246 to retrieve the encoded data components in a mannersimilar to that discussed in connection with step 614 of process 600described above.

In some embodiments, process 700 includes causing 716 the deletion of atleast some of the data components. For example, in an environment 200illustrated by FIG. 2, a storage node manager 244 responsible for thedata deletion job may identify a set of storage nodes that store thedata components for the data to be deleted and requests at least asubset of those storage nodes to delete their respective datacomponents. Each of the subset of storage node 246, upon receiving arequest from storage node manager 244 to delete a data component, mayvalidate the request, for example, by checking the integrity of astorage location identifier (that is part of the data objectidentifier), verifying that the storage node indeed holds the requesteddata component and the like. Upon a successful validation, the storagenode may delete the data component from a connected storage device andsends an acknowledgement to storage node manager 244 indicating whetherthe operation was successful. In an embodiment, multiple data deletionjobs may be executed in a batch such that data objects located closetogether may be deleted as a whole. In some embodiments, data deletionis considered successful when storage node manager 244 receives positiveacknowledgement from at least a subset of storage nodes 246. The size ofthe subset may be configured to ensure that data cannot be reconstructedlater on from undeleted data components. Failed or incomplete datadeletion jobs may be retried in a manner similar to the retrying of datastorage jobs and data retrieval jobs, described in connection withprocess 500 and process 600, respectively.

In an embodiment, process 700 includes updating 718 metadata informationsuch as that described in connection with step 526 of process 500. Forexample, storage nodes executing the deletion operation may updatestorage information including index, free space information and thelike. In an embodiment, storage nodes may provide updates to storagenode registrar or storage node registrar store. In various embodiments,some of such metadata information may be updated via batch processingand/or on a periodic basis to reduce performance and cost impact.

FIG. 8 depicts an illustrative data structure 800 in which additionaltechniques for the validation of data integrity may be implemented.Illustrative data structure 800 is but one of many different types ofdata structures that may be utilized to implement the techniquesdescribed herein. By way of example only, a user or client entity maywish to upload a data payload 802 to the archival data storage service.The archival data storage service may then be configured to receive thedata payload 802 (in one or more parts) and allow the user to verify, atsome point (e.g., immediately after upload or after some time, in somecases, after a relatively long time), that the data stored in thearchival data storage service is, in fact, the same as the data payload802 that was uploaded without requesting any size partitioninginformation from the user. In other words, the archival data storageservice may provide a data object identifier that the user may return inorder to retrieve stored data; however, the user may not need to storeany information other than the data object identifier.

In some examples, in order to accept data from the user, the archivaldata storage service may request that the user provide a tree digestlike the data structure 800 of FIG. 8. Providing the data structure 800may be performed in multiple ways in accordance with variousembodiments. For example, all of the data illustrated in the datastructure 800 may be provided. As an alternative, in embodiments wherethe data structure 800 is constructible solely from the data for theleaf nodes, data for the leaf nodes may be provided without providinginformation for other, higher-level nodes. Additionally, the archivaldata storage service may provide instructions in the form of analgorithm, API and/or SDK for generating the data structure 800. In someinstances, limitations on the size of upload chunks and their respectiveoffsets may be imposed. For example, the chunks or parts of the datapayload 802 may be limited to powers of two of 1 MB. Additionally, insome examples, the determined size of each chunk may not be changedwithin a particular upload. Further, for each part received, thearchival data storage service may calculate its own digest, based atleast in part on the same algorithm used by the user, and provide thedigest for each part. Upon completion of the storage job, the archivaldata storage service may provide the top-level digest value in the formof a data object identifier. Retrieval of the data may, in someexamples, be implemented in a similar fashion, with restrictions onchunk sizes and offsets limited to powers of two by 1 MB, messagesprepended with the digest of the data that is in the message and thetop-level digest available upon completion of the job. However, based atleast in part on this implementation, the data payload 802 should beable to be verified or validated independent of the chunk size selectedby the user. A digest may be calculated by applying a cryptographic hashfunction such as those associated with SHA-1, SHA-2, MD5, MD6 and thelike, a checksum or error-detection code such as cyclic redundancy checkand the like to at least a portion of the payload data.

The data structure 800 of FIG. 8 may illustrate an appropriate digesttree for a data payload 802 where the user has chosen to upload the datapayload in a single part. As such, there is no part size for the user toselect in this example. However, the resulting root digest 806 should becalculable using the techniques described herein even if the user hadselected to upload the data payload 802 in multiple parts, and even ifthe user had selected a part size unknown to the archival data storageservice and/or not recorded by the user. In this example, for the sakeof simplicity, it will be assumed that the data payload 802 is 7 MBs insize. As such, and since the user has requested to upload the entirepayload 802 in one part, the data payload 802 may be partitioned intoseven 1 MB chunks, Sub 1-Sub 7. In some examples, however, if the sizeof the payload 802 were not divisible by 1 MB, the last chunk, Sub 7,may be smaller than 1 MB. The archival data storage service may, basedat least in part on the hash tree algorithm, generate a hash value (ordigest) for each 1 MB chunk (i.e., Sub 1-Sub 7). Each of these hashvalues may be represented at the lowest child node level 808 of the datastructure 800. In order to generate the nodes of the second child nodelevel 810, the archival data storage service may concatenate each pairof second-level node children and run the hash function on theconcatenated data. In other words, the lowest level 808 of the datastructure may include a digest of payload data, while parent nodes mayinclude digests of digests. Moving up the data structure, the describedoperations may be repeated until a root digest 806 is generated.

As described, in some cases, the archival data storage service mayprovide intermediary root digests for individual parts of the payload802. However, in this example, since the payload was not broken intoparts, the archival data storage service may only provide the rootdigest 806 to the user. In some cases, though, the archival data storageservice may also provide each 1 MB digest generated. As such, either theuser or the archival data storage service should be able to verify thatthe data was uploaded correctly (including at the 1 MB sub-part level)based at least in part on comparing each other's generated root digest806.

FIG. 9 depicts another illustrative data structure 900 in whichadditional techniques for the validation of data integrity may beimplemented. As noted with reference to FIG. 8, the illustrative datastructure 900 is but one of many different types of data structures thatmay be utilized to implement the techniques described herein. By way ofexample only, a user or client entity may wish to upload a data payload902 to the archival data storage service. The archival data storageservice may then be configured to receive the data payload 902 (in thisexample, in two parts) and allow the user to verify that the data storedin the archival data storage service is, in fact, the same as the datapayload 902 that was uploaded. This validation may be done withoutrequesting any size partitioning information from the user. In otherwords, the archival data storage service may provide a data objectidentifier that the user may return in order to retrieve stored data;however, the user may not need to store any information other than thedata object identifier in order to request and/or validate the storeddata.

In generating the data structure 900, the user or the archival datastorage service may once again break the data into sub-parts; however,in this example, each part Part 1 or Part 2 may be broken up separately(e.g., Sub 1-Sub 4 of Part 1 and Sub 1-Sub 3). Again, a digest for eachsub-part may be generated and included in the data structure at thechild level 904 and digests of concatenated digests may be generated andincluded in the data structure at a first parent level 906. In thisexample, however, since the payload 902 has been broken into two parts,a top-level digest may be generated for each part. As such, Part 1digest 908 and Part 2 digest 910 may be generated and included in thedata structure 900. Additionally, as the payload 902 is uploaded, eachof the sub-part digests (e.g., those at 904) and the part digests (e.g.,those at 908) may be included in the upload. Further, a root digest 912may be generated in the same fashion that the other parent nodes aregenerated. That is, based at least in part on concatenating the childrendigests, and running the hash function on the concatenated information.In this example, this process would entail concatenating Part 1 digest908 and Part 2 digest 910 to generate a part-level digest. The archivaldata storage service may then run the hash function on the part-leveldigest to generate the root digest 912. In some examples, the rootdigest may be received at the beginning of upload and once the upload iscompleted. Additionally, the archival data storage service may generateits own version of the data structure 900 and/or the root digest 912 inorder to validate the integrity of the data. Further, in some examples,the root digest 912 generated by the archival data storage service maybe provided to the user as part of a data object identifier that theuser may utilize to make read, delete or index viewing requests.

In some examples, the archival data storage service may assume that datacorruptions can occur anywhere in the system and/or may be caused byhardware bugs, bit flips and/or due to bugs in software code implementedby the archival data storage service or the user. For at least thisreason, the archival data storage service may review all, or a subset,of the data paths and operations to ensure that data integrity isprovided throughout the system and that corrupt data is detected. Insome cases, this may apply to the data payload (e.g., that stored in thearchival data storage service) and to the metadata. As such, dataintegrity validation may be performed on the data object identifiers aswell to ensure that requests to delete data are not pointing at thewrong data.

In some aspects, the archival data storage service 206 may be configuredto expect that a selected or otherwise determined digest function may beacceptable for the validation of data integrity. In some examples, thedigest function may not be used for some cases related to datatransformation. Otherwise, it may be selected and/or provided for usewith validating some, al, or portions of the data and/or metadata of thearchival data storage service. Additionally, as noted, in some examples,the initiator (i.e., the user) may pre-calculate the digest of the datapayload 902 before transmission and then later may supply the digestagain with the data to the archival data storage service. The archivaldata storage service may then recalculate the digest (e.g., thetop-level digest), compare with the digest received from the initiatorand/or acknowledge that the integrity of the data was validated byproviding the archival data storage service-generated digest to theuser. Additionally, each data subdivision and/or aggregation (e.g., thesub-parts, the parts, the part-level digests and/or the root digests)may be re-validated by calculating the independent digest on the splitof the aggregate data and comparing the digest or even by performing abit by bit comparison. In other words, given any data payload, of anysize, calculations may be performed to generate any number or type ofthe split or aggregated digests and, thus, validate the data and/or theparts.

Additionally, in some aspects, data transformation such as, but notlimited to, erasure coding or encryption can be re-validated byperforming the reverse transformation. The results of the reversetransformation may then be cross checked by comparing the digest and/orby bit by bit comparison. As such, the transformed data may include twodigests. One of the two may testify to the integrity of the transformeddata and the other may testify to the integrity of the original data. Insome examples, referential items such as, but not limited to, the dataobject identifier that may reference the content may include the digestof the data being referenced. Additionally, the archival data storageservice may also include information about the parent node that is beingreferenced. In some cases, messages from the control plane that arepersisted in the storage node registrar store 250, the data objectidentifier and/or other data structures may include digests thatself-validate. These digests may be produced after the structures arecreated and/or verified upon retrieval or before the action. In someexamples, this prevent things such as bugs in the code, memorycorruption or bit rot from flipping data object retrieve commands intodelete commands. Further, on the return path, the archival data storageservice may be configured to re-validate that the data that is beingreturned to the customer is matched against the request and/or that nosubstitution during execution happens due to a bug in the code or otherthe like.

FIGS. 10-12 illustrate example flow diagrams showing respectiveprocesses 1000-1200 for providing validation of data integrity. Theseprocesses are illustrated as logical flow diagrams, each operation ofwhich represents a sequence of operations that can be implemented inhardware, computer instructions, or a combination thereof. In thecontext of computer instructions, the operations representcomputer-executable instructions stored on one or more computer-readablestorage media that, when executed by one or more processors, perform therecited operations. Generally, computer-executable instructions includeroutines, programs, objects, components, data structures and the likethat perform particular functions or implement particular data types.The order in which the operations are described is not intended to beconstrued as a limitation, and any number of the described operationscan be combined in any order and/or in parallel to implement theprocesses.

Additionally, some, any, or all of the processes may be performed underthe control of one or more computer systems configured with executableinstructions and may be implemented as code (e.g., executableinstructions, one or more computer programs, or one or moreapplications) executing collectively on one or more processors, byhardware, or combinations thereof. As noted above, the code may bestored on a computer-readable storage medium, for example, in the formof a computer program comprising a plurality of instructions executableby one or more processors. The computer-readable storage medium may benon-transitory.

In some aspects, the API request handler 218, the payload data cache228, the storage node manager 244 and/or the storage nodes 246 of theone or more archival data storage service 206 shown in FIG. 2 mayperform the process 1000 of FIG. 10. The process 1000 may begin byproviding (e.g., to a user or client entity of the archival data storageservice) a function call for requesting data storage at 1002. Thefunction call may be part of an API or an SDK for interacting withand/or interfacing with the archival data storage service. At 1004, theprocess 1000 may include receiving a plurality of portions of a datapayload from a remote computing device (i.e., the user). In some cases,the size of each portion may be consistent. In other cases, the size ofeach portion may be consistent except that the last portion may bedifferent. Additionally, the size may be selected or otherwisedetermined by the user. At 1006, the process 1000 may include receivingan indication of the size of each portion. In some instances, theactions performed at 1004 and 1006 may be performed together as a singleaction. However, some restrictions may apply regarding portion size.Such as, in some examples, the portions may be limited to a consistentsize (i.e., they may be required to be the same size); however, the lastportion may be a remainder of the data payload (i.e., the payload minuseach other consistently-sized portion). For example, the size selectionmay be limited to 1 MB or an integer multiple of 1 MB. In otherexamples, the size may be limited to 1 MB or a power of two of 1 MB.

The process 1000 may also include generating one or more sub-portions ofa predefined size for at least some of the portions of the payload at1008. As noted above, regarding the portion size, while sub-portion sizemay be predefined and constant, the last sub-portion may be a differentsize. The predefined size may be 1 MB or any other size. At 1010, theprocess 1000 may include calculating one or more digests or hash valuesbased at least in part on the sub-portions. The digests may becalculated based at least in part on a published and/or otherwiseprovided algorithm. In some examples, at 1012, the process 1000 mayinclude generating a root node of a data structure. The data structuremay be based at least in part on the sub-portion digests and/oraggregated digests as described above. At 1014, the process 1000 mayinclude determining a top-level digest of the data structure. Thetop-level digest may be based at least in part on the root node of thedata structure and/or on a parent node associated with one of theportions of data. At 1016, the process 1000 may include providinginstructions configured to enable the remote computing device togenerate the data structure. In this way, the user may generate the datastructure along with the archival data storage service. The process 1000may then include receiving a top-level digest generated by the remotecomputing device at 10108. The process 1000 may end at 1020 by verifyingthat the stored data payload matches a received data payload. In otherwords, the process 1000 may validate or verify the integrity of thedata.

FIG. 11 illustrates another example flow diagram showing process 1100for validating the integrity of data. In some aspects, the API requesthandler 218, the payload data cache 228, the storage node manager 244and/or the storage nodes 246 of the one or more archival data storageservice 206 shown in FIG. 2 may perform the process 1100 of FIG. 11. Theprocess 1100 may begin by receiving one or more parts of a data payloadat 1102. As noted above, the parts may be any size. However, in someexamples, the part size may be limited to 1 MB or multiples of 1 MB. Inthis way, a data structure may be composed independent of the chosensize. At 1104, the process 1100 may include generating a sub-part forthe one or more parts. Again, these sub-parts may be any size or may belimited to 1 MB or other size limitation such as, but not limited to, 2MB, LOMB, etc. the process 1100 may include calculating a value based onthe sub-part at 1106. The value may, in some cases, be a hash value, adigest, or other result of encryption. In some examples, the process1100 may include generating a root node of a data structure at 1108. At1110, the process 1100 may include determining a top-level value of thedata structure based at least in part on traversing the data structureto the root node.

In some examples, the process 1100 may also include storing the datapayload at 1112. The payload may be stored based at least in part oncombining each of the one or more parts received at 1102. As such, insome cases, the archival data storage service 206 may not be able tostore the payload at 1112 until the data transmission of all the partsis complete. At 1114, the process 1100 may include validating that thestored data payload matches the received data payload. This may beperformed by comparing the received top-level value with a calculatedtop-level value. At 1116, the process 1100 may include providing a dataobject identifier including the top-level value. The identifier maylater be utilized by the user to retrieve and/or delete the stored datapayload. In some examples, the process 1100 may include receiving arequest for the stored payload at 1118. The stored payload may beprovided back to the user in a similar fashion (with the integrity ofthe data being validated each step of the way). However, in some cases,the process 1100 may end at 1120 by verifying that the stored datapayload has not changed prior to providing the payload.

FIG. 12 illustrates another example flow diagram showing process 1200for validating the integrity of data. In some aspects, the API requesthandler 218, the payload data cache 228, the storage node manager 244and/or the storage nodes 246 of the one or more archival data storageservice 206 shown in FIG. 2 may perform the process 1200 of FIG. 12. Theprocess 1200 may begin by providing instructions for making method callsto perform operations on data at 1202. In some examples, these methodcalls may be exposed via one or more APIs or provided in one or moreSDKs. At 1204, the process 1200 may include performing a firstoperation, using a verification algorithm, based on a first partitioningof a data object into first partitions. The first partitions, in someexamples, may include 1 MB or other consistent sized chunks that may beutilized to generate a data structure such as, but not limited to, ahash tree or other binary tree of digests. In some examples, the firstoperation may include receiving the data from a user over a network. At1206, the process 1200 may include verifying the data object to generatea first verification value (e.g., a hash code, checksum, etc.) based onthe first partitions. The process 1200 may also include performing asecond operation on a data object, utilizing the same verificationalgorithm, based at least in part on a second partitioning of the dataobject into second partitions at 1208. The second partitions may be adifferent size from the first partitions. Based at least in part on thesecond partitions, the process 1200 may include verifying the dataobject to generate a second value at 1210. Here, the second operationmay also include transmitting data to the archival data storage service.The second verification value, like the first may include, but is notlimited to, a digest for a partition, a digest for digests formed byaggregating partition digests and/or a top-level digest of a datastructure. At 1212, the process 1200 may end by determining whether thesecond verification value equals the first verification value. This maybe determined based at least in part on comparing the two values. Insome examples, if the verification algorithm is properly performed, andthe data has maintained its integrity, the two values are expected to beequal. That is, independent of the size of the two sets of partitions(i.e., the first partitioning and the second partitioning), theverification values should be equal.

Illustrative methods and systems for validating the integrity of dataare described above. Some or all of these systems and methods may, butneed not, be implemented at least partially by architectures such asthose shown above.

FIG. 13 illustrates aspects of an example environment 1300 forimplementing aspects in accordance with various embodiments. As will beappreciated, although a Web-based environment is used for purposes ofexplanation, different environments may be used, as appropriate, toimplement various embodiments. The environment includes an electronicclient device 1302, which can include any appropriate device operable tosend and receive requests, messages or information over an appropriatenetwork 1304 and convey information back to a user of the device.Examples of such client devices include personal computers, cell phones,handheld messaging devices, laptop computers, set-top boxes, personaldata assistants, electronic book readers and the like. The network caninclude any appropriate network, including an intranet, the Internet, acellular network, a local area network or any other such network orcombination thereof. Components used for such a system can depend atleast in part upon the type of network and/or environment selected.Protocols and components for communicating via such a network are wellknown and will not be discussed herein in detail. Communication over thenetwork can be enabled by wired or wireless connections and combinationsthereof. In this example, the network includes the Internet, as theenvironment includes a Web server 1306 for receiving requests andserving content in response thereto, although for other networks analternative device serving a similar purpose could be used as would beapparent to one of ordinary skill in the art.

The illustrative environment includes at least one application server1308 and a data store 1310. It should be understood that there can beseveral application servers, layers, or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. As used herein the term “data store” refers to any device orcombination of devices capable of storing, accessing and retrievingdata, which may include any combination and number of data servers,databases, data storage devices and data storage media, in any standard,distributed or clustered environment. The application server can includeany appropriate hardware and software for integrating with the datastore as needed to execute aspects of one or more applications for theclient device, handling a majority of the data access and business logicfor an application. The application server provides access controlservices in cooperation with the data store, and is able to generatecontent such as text, graphics, audio and/or video to be transferred tothe user, which may be served to the user by the Web server in the formof HTML, XML or another appropriate structured language in this example.The handling of all requests and responses, as well as the delivery ofcontent between the client device 1302 and the application server 1308,can be handled by the Web server. It should be understood that the Weband application servers are not required and are merely examplecomponents, as structured code discussed herein can be executed on anyappropriate device or host machine as discussed elsewhere herein.

The data store 1310 can include several separate data tables, databasesor other data storage mechanisms and media for storing data relating toa particular aspect. For example, the data store illustrated includesmechanisms for storing production data 1312 and user information 1316,which can be used to serve content for the production side. The datastore also is shown to include a mechanism for storing log data 1314,which can be used for reporting, analysis or other such purposes. Itshould be understood that there can be many other aspects that may needto be stored in the data store, such as for page image information andto access right information, which can be stored in any of the abovelisted mechanisms as appropriate or in additional mechanisms in the datastore 1310. The data store 1310 is operable, through logic associatedtherewith, to receive instructions from the application server 1308 andobtain, update or otherwise process data in response thereto. In oneexample, a user might submit a search request for a certain type ofitem. In this case, the data store might access the user information toverify the identity of the user, and can access the catalog detailinformation to obtain information about items of that type. Theinformation then can be returned to the user, such as in a resultslisting on a Web page that the user is able to view via a browser on theuser device 1302. Information for a particular item of interest can beviewed in a dedicated page or window of the browser.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server, and typically will include a computer-readablestorage medium (e.g., a hard disk, random access memory, read onlymemory, etc.) storing instructions that, when executed by a processor ofthe server, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available, and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment in one embodiment is a distributed computing environmentutilizing several computer systems and components that areinterconnected via communication links, using one or more computernetworks or direct connections. However, it will be appreciated by thoseof ordinary skill in the art that such a system could operate equallywell in a system having fewer or a greater number of components than areillustrated in FIG. 13. Thus, the depiction of the system 1300 in FIG.13 should be taken as being illustrative in nature, and not limiting tothe scope of the disclosure.

The various embodiments further can be implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers, computing devices or processing devices which can be used tooperate any of a number of applications. User or client devices caninclude any of a number of general purpose personal computers, such asdesktop or laptop computers running a standard operating system, as wellas cellular, wireless and handheld devices running mobile software andcapable of supporting a number of networking and messaging protocols.Such a system also can include a number of workstations running any of avariety of commercially-available operating systems and other knownapplications for purposes such as development and database management.These devices also can include other electronic devices, such as dummyterminals, thin-clients, gaming systems and other devices capable ofcommunicating via a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TCP/IP, OSI, FTP,UPnP, NFS, CIFS and AppleTalk. The network can be, for example, a localarea network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network and any combination thereof.

In embodiments utilizing a Web server, the Web server can run any of avariety of server or mid-tier applications, including HTTP servers, FTPservers, CGI servers, data servers, Java servers and businessapplication servers. The server(s) also may be capable of executingprograms or scripts in response requests from user devices, such as byexecuting one or more Web applications that may be implemented as one ormore scripts or programs written in any programming language, such asJava®, C, C# or C++, or any scripting language, such as Perl, Python orTCL, as well as combinations thereof. The server(s) may also includedatabase servers, including without limitation those commerciallyavailable from Oracle®, Microsoft®, Sybase® and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (“SAN”) familiar to those skilledin the art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (CPU), at least one inputdevice (e.g., a mouse, keyboard, controller, touch screen or keypad),and at least one output device (e.g., a display device, printer orspeaker). Such a system may also include one or more storage devices,such as disk drives, optical storage devices, and solid-state storagedevices such as random access memory (“RAM”) or read-only memory(“ROM”), as well as removable media devices, memory cards, flash cards,etc.

Such devices also can include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device, etc.) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium, representing remote, local, fixed and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs, such as a client applicationor Web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, EEPROM, flash memoryor other memory technology, CD-ROM, digital versatile disk (DVD) orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices or any other medium which canbe used to store the desired information and which can be accessed bythe a system device. Based on the disclosure and teachings providedherein, a person of ordinary skill in the art will appreciate other waysand/or methods to implement the various embodiments.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

Other variations are within the spirit of the present disclosure. Thus,while the disclosed techniques are susceptible to various modificationsand alternative constructions, certain illustrated embodiments thereofare shown in the drawings and have been described above in detail. Itshould be understood, however, that there is no intention to limit theinvention to the specific form or forms disclosed, but on the contrary,the intention is to cover all modifications, alternative constructionsand equivalents falling within the spirit and scope of the invention, asdefined in the appended claims.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosed embodiments (especially in thecontext of the following claims) are to be construed to cover both thesingular and the plural, unless otherwise indicated herein or clearlycontradicted by context. The terms “comprising,” “having,” “including,”and “containing” are to be construed as open-ended terms (i.e., meaning“including, but not limited to,”) unless otherwise noted. The term“connected” is to be construed as partly or wholly contained within,attached to, or joined together, even if there is something intervening.Recitation of ranges of values herein are merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range, unless otherwise indicated herein, and eachseparate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g., “such as”) provided herein, isintended merely to better illuminate embodiments of the invention anddoes not pose a limitation on the scope of the invention unlessotherwise claimed. No language in the specification should be construedas indicating any non-claimed element as essential to the practice ofthe invention.

Preferred embodiments of this disclosure are described herein, includingthe best mode known to the inventors for carrying out the invention.Variations of those preferred embodiments may become apparent to thoseof ordinary skill in the art upon reading the foregoing description. Theinventors expect skilled artisans to employ such variations asappropriate, and the inventors intend for the invention to be practicedotherwise than as specifically described herein. Accordingly, thisinvention includes all modifications and equivalents of the subjectmatter recited in the claims appended hereto as permitted by applicablelaw. Moreover, any combination of the above-described elements in allpossible variations thereof is encompassed by the invention unlessotherwise indicated herein or otherwise clearly contradicted by context.

All references, including publications, patent applications and patents,cited herein are hereby incorporated by reference to the same extent asif each reference were individually and specifically indicated to beincorporated by reference and were set forth in its entirety herein.

1. A computer-implemented method for data integrity verification,comprising: under control of one or more computer systems configuredwith executable instructions, receiving one or more parts of a datapayload; for each part of at least a subset of the one or more parts,generating a first sub-part as part of partitioning the data payload;calculating a value based at least in part on the first generatedsub-part; generating at least a root node of a data structure based atleast in part on the calculated value; identifying a top-level value ofthe data structure associated with the root node of the data structure;and verifying, based at least in part on the top-level value, that arepartitioned data payload matches the received data payload, therepartitioned data payload based at least in part on a second sub-partof the one or more parts.
 2. The computer-implemented method of claim 1,further comprising providing an instruction to a remote computingdevice, the instruction configured to enable the remote computing deviceto generate the data structure and provide the identified top-levelvalue.
 3. The computer-implemented method of claim 1, wherein the one ormore parts of the data payload are received as part of an electronicrequest, from a remote computing device, to store the data payload. 4.The computer-implemented method of claim 3, further comprising providinga function call for a remote device to make the electronic request, andwherein the electronic request includes at least the provided functioncall.
 5. The computer-implemented method of claim 1, wherein the datastructure includes at least a binary tree.
 6. The computer-implementedmethod of claim 1, further comprising receiving an indication of a sizefor each of the one or more parts of the data payload.
 7. Acomputer-implemented method for data integrity verification, comprising:under control of one or more computer systems configured with executableinstructions, receiving one or more parts of a data payload; for eachpart of at least a subset of the one or more parts, generating asub-part of a first predefined size, the received one or more partsbeing a second size that is an integer multiple of the first predefinedsize; calculating a value based at least in part on the generatedsub-part; generating at least a root node of a data structure based atleast in part on the calculated value; and determining a top-level valueof the data structure associated with the root node of the datastructure.
 8. The computer-implemented method of claim 7, furthercomprising: storing the data payload in archival data storage; andvalidating, based at least in part on the determined top-level value anda received top-level value, that the stored data payload matches areceived data payload corresponding to the received one or more parts.9. (canceled)
 10. The computer-implemented method of claim 7, whereinthe integer is based at least in part on a degree of branches of thedata structure.
 11. The computer-implemented method of claim 7, furthercomprising providing a data object identifier for identifying the datapayload, the data object identifier including the top-level value of thedata structure.
 12. The computer-implemented method of claim 11, furthercomprising: receiving a request for the stored data payload; andverifying, based at least in part on the top-level value, that thestored data payload has not changed prior to providing the data payload.13. The computer-implemented method of claim 7, further comprising:storing the data payload of a first storage device to second storagedevice; and verifying, based at least in part on the top-level value,that the data payload stored in the second storage device matches thedata payload of the first storage device.
 14. The computer-implementedmethod of claim 7, further comprising: repartitioning the data payloadbased at least in part on a second generated sub-part, the secondgenerated sub-part being different from the generated sub-part; andverifying, based at least in part on the top-level value, that therepartitioned data payload matches a stored data payload, the storeddata payload based at least in part on the received data payload.
 15. Asystem for verifying data integrity, comprising: at least one memorythat stores computer-executable instructions; and at least one processorconfigured to access the at least one memory, wherein the at least oneprocessor is configured to execute the computer-executable instructionsto collectively at least: store at least a chunk of data, the datacomprising a plurality of chunks; generate a first sub-chunk of the atleast one chunk of data; store at least a digest corresponding to thefirst sub-chunk; generate a data structure based at least in part on thestored digest corresponding to the first sub-chunk; provide a top-leveldigest of the generated data structure to a computing device, andverify, based at least in part on the top-level digest, that arepartitioned chunk of data matches the stored at least a chunk of data,the repartitioned chunk of data based at least in part on a secondsub-chunk of the one or more parts.
 16. The system of claim 15, whereinthe at least one processor is further configured to execute thecomputer-executable instructions to receive the at least a chunk of datafrom the computing device based at least in part on a predefined sizeindicated by the computing device.
 17. The system of claim 16, whereinthe at least a chunk of data is received in an electronic request, fromthe computing device, to store the data based at least in part on aninstruction format provided by the system.
 18. The system of claim 15,wherein the top-level digest is prepended to an identifier configured toidentify the data.
 19. The system of claim 18, wherein the at least oneprocessor is further configured to execute the computer-executableinstructions to: store the data comprising the plurality of chunks;receive a request to provide the data to the computing device, therequest including the identifier; and validate the data, based at leastin part on the top-level digest, prior to providing the data to thecomputing device.
 20. The system of claim 18, wherein the at least oneprocessor is further configured to execute the computer-executableinstructions to: generate a digest for the identifier; and validate theidentifier, based at least in part on the generated digest for theidentifier, prior to referencing the identifier in an instruction. 21.One or more computer-readable media storing computer-executableinstructions for verifying data integrity that, when executed by one ormore processors, configure the one or more processors to performoperations comprising: performing a first data operation in connectionwith a data object, the first data operation based at least in part on afirst partitioning of the data object into first partitions; verifyingthe data object using a data verification algorithm to generate a firstverification value based at least in part on the first partitions bygenerating one or more partitions of the data in a predetermined size;performing a second data operation in connection with the data object,the second data operation based at least in part on a secondpartitioning of the data object into second partitions, the secondpartitions being different from the first partitions and at least one ofthe first partitions or at least one of the second partitions having asize that is a multiple of the predetermined size; verifying the dataobject using the data verification algorithm to generate a secondverification value; and determining whether the second verificationvalue matches the first verification value.
 22. (canceled)
 23. The oneor more computer-readable media of claim 21, wherein the multiple is aresult of integer exponentiation of a degree of a data structureassociated with the first data object.
 24. The one or morecomputer-readable media of claim 21, wherein the instructions furtherconfigure the one or more processors to perform operations comprisingproviding instructions for making method calls to perform at least oneof the first data operation or the second data operation.
 25. The one ormore computer-readable media of claim 21, wherein at least one of thefirst data operation or the second data operation is a transfer of thedata object over a network.
 26. The one or more computer-readable mediaof claim 25, wherein the object is transferred over the network inparts.
 27. The one or more computer-readable media of claim 21, whereinthe first operation includes at least receiving the data object and thesecond operation includes at least storing the data object in archivaldata storage.
 28. The one or more computer-readable media of claim 27,wherein the instructions further configure the one or more processors toperform operations comprising ensuring integrity of the data objectbased at least in part on the second verification value.
 29. The one ormore computer-readable media of claim 27, wherein the instructionsfurther configure the one or more processors to perform operationscomprising ensuring integrity of the data object based at least in parton the second partitions.